01
Critical
Example: The participant cannot finish the task, or makes an error that could cause harm, such as entering a wrong dose.
Approach: Fix before release, then retest with new participants.
Usability Testing
Usability testing services
Teams often learn that users cannot finish a task only after launch, from support tickets. Agnotic Technologies runs usability tests before release, with the right study type, real participants and findings rated by severity, then retests the fixes.
Moderated and remoteAssistive technology usersSeverity-rated findingsHuman factors studies
Trusted by global innovators
What is usability testing?
Usability testing is watching real users try to complete set tasks with your product, so you can see where they fail before release. Agnotic Technologies plans and runs moderated, remote and assistive technology studies for web and mobile products, and supports the formative and summative human factors testing medical device software needs.
ISO 9241-11 defines usability as the extent to which specified users can achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. A test measures exactly that, one task at a time.
Wider research, such as interviews and surveys, sits in our UX research service. This page covers the test itself.
Decision table
Which usability study fits your product and stage?
The study type decides who takes part, what you learn and whether the result can serve as regulatory evidence.
| Study type | When it fits | Participants | Output | Counts as FDA human factors evidence? |
|---|---|---|---|---|
| Formative moderated test | Early designs and prototypes, while changes are cheap | Small rounds from each user group | Observed problems and design changes | Supports the design process; not final validation |
| Unmoderated remote test | Specific tasks at larger scale on a staging or live build | Recruited panel or your own users | Task success, time on task and recordings | No |
| Accessibility test with assistive technology users | Before release and after major interface changes | People who use screen readers, magnification, switch access or voice control | Barriers by task, linked to WCAG criteria | No, unless planned into a validation study |
| Summative human factors validation | Final, production-equivalent medical device software | Representative users from each intended user group, in realistic conditions | Validation report on critical tasks | Yes, when planned and run to FDA guidance and IEC 62366-1 |
| Safety-enhanced design test for certified health IT | EHR capabilities going through ONC certification | At least 10 participants per capability | Summative test report for certification | No; it is ONC certification evidence |
Most products need two or three formative rounds before any summative study. Skipping them usually means the summative study finds problems that are expensive to fix.
How we run it
How do we run a usability test?
The steps behind every study, from the first question to the retest.
Agree the questions
Which decisions the test should inform, which user groups matter and which tasks are critical.
Write task scripts
Realistic tasks with a starting point, the information a user would have and a clear definition of success.
Recruit participants
People who match each user group, including people who use assistive technology, screened before sessions.
Run the sessions
A moderator and a note taker, or an unmoderated tool for larger samples, with recordings where participants agree.
Rate the findings
Each problem is rated by severity, linked to the task and illustrated with what participants did and said.
Fix and retest
After your team changes the design, we retest the failed tasks with new participants.
Severity
How do we rate usability findings?
A shared scale, so your team fixes the right problems first.
02
Serious
Example: The participant finishes only with help or after several failed attempts.
Approach: Fix in the current release cycle.
03
Minor
Example: The participant hesitates or takes a longer path but succeeds alone.
Approach: Add to the backlog and group with related fixes.
04
Observation
Example: A preference or suggestion with no effect on task success.
Approach: Record it, and act if it repeats across participants.
Medical device software
What does medical device software need from human factors testing?
For software that is a medical device, usability testing is part of the safety case, not only a design check.
- A use-related risk analysis that lists the critical tasks and links each one to the device risk file
- Formative evaluations during design, recorded so the design history shows how problems were found and fixed
- A summative validation study with representative users from each user group, on a production-equivalent build
- Realistic use conditions, such as interruptions, glare or gloves where the real setting has them
- A process that follows IEC 62366-1 and the FDA's human factors guidance for medical devices
- Your regulatory team owns the submission; we plan and run the studies and write the reports they need
Who it is for
Who do we run usability tests for?
Teams that want to see real users work through their product before customers do.
Usability testing for startups
Founders checking that first users can complete the core task in an MVP before spending on growth.
SaaS and web product teams
Teams testing a new flow, a redesign or an onboarding change before release.
Mobile app teams
Tests on real devices, including one-handed use, small screens and interrupted sessions.
Clinical software teams
Tests with clinicians and patients, scheduled around clinical work.
Medical device software makers
Formative rounds and summative human factors validation planned with your regulatory team.
Standards
Standards our usability studies follow
The definitions and methods behind the study types above.
ISO 9241-11IEC 62366-1FDA human factors45 CFR 170.315(g)(3)NISTIR 7804WCAG 2.2
Facts and sources
Facts behind usability testing
Standards and rules that shape a usability study, each linked to its primary source.
IEC 62366-1:2015 specifies a process for a medical device manufacturer to analyse, specify, develop and evaluate the usability of a device as it relates to safety, including risks from use errors.
Source: IEC 62366-1:2015 (IEC Webstore)ONC's safety-enhanced design criterion requires user-centered design for listed certified health IT capabilities, with at least 10 test participants for the testing of each capability.
Source: 45 CFR 170.315(g)(3) (eCFR)NIST's EHR Usability Protocol (NISTIR 7804) combines an expert review of the interface with validation studies, meaning summative usability tests with representative users on realistic tasks.
Source: NISTIR 7804 (NIST)The W3C Web Accessibility Initiative advises that involving people with disabilities throughout development is more effective than only running formal usability testing at the end of a project.
Source: Involving Users in Evaluating Web Accessibility (W3C WAI)
Scope and cost
What drives the cost of usability testing?
Studies are quoted individually. These are the factors that change their size.
User groups
How many distinct user groups need their own participants and tasks.
Recruiting
How hard participants are to find, such as clinicians, specialists or assistive technology users.
Study type
Moderated sessions, unmoderated remote tests, or a summative validation study.
Setting
Remote sessions, your office, or a simulated real-world environment.
Test material
Whether you have a clickable prototype or a build, or one needs to be made first.
Reporting depth
A findings report for the design team, or validation documentation for a regulatory file.
Frequently Asked Questions
It depends on the study. Formative tests run in small rounds per user group and repeat after each fix. Summative studies need more participants per user group, and ONC's safety-enhanced design criterion sets a minimum of 10 per capability.
In a moderated test a facilitator watches each session and can ask why a participant did something. An unmoderated test runs through a tool without a facilitator, which reaches more people but explains less.
Yes. We recruit people who use screen readers, magnification, switch access or voice control, and plan sessions around their own devices and settings, so findings reflect how they really work.
Yes. A clickable prototype is enough for a formative test, and testing at that stage is the cheapest time to find problems. Summative studies need a production-equivalent build.
We plan and run formative and summative studies to IEC 62366-1 and FDA human factors guidance, and write the reports. Your regulatory team decides the submission strategy and owns the submission.
UX research covers everything you do to understand users, including interviews and surveys. Usability testing is one method within it: observing people perform set tasks with your product.
Each study is quoted on its own. Recruiting is often the biggest factor, alongside the number of user groups, the study type, the setting and whether validation documentation is needed.