User research is increasingly used to justify how government and business invest in digital services, yet much of that evidence rests on usability tests with a handful of participants. User experience research is a young profession, and many practitioners enter it through short courses that teach interviewing and testing techniques, but not statistics or research design. As a result, small tests are routinely used to support claims they cannot bear: that one design is better than another, that a change will improve satisfaction by a given amount, or that a problem affects most users. When so few participants can be made to support almost any conclusion, decisions worth millions of dollars end up resting on little more than the judgement of whoever ran the test.
The practice of testing with small samples has respectable origins. In 1993, Jakob Nielsen and Thomas Landauer showed that the discovery of usability problems follows a predictable pattern, which they modelled as a Poisson process. Across the studies they examined, a single test user uncovered on average around 31% of a design’s usability problems, so five users together would be expected to find roughly 85% of them. This makes a strong case for iterative design, where a small group tests a design, the team fixes what they find, and a new group tests again. Used in this way, small samples are efficient and entirely defensible.
The difficulty is that this model answers only one question, which is whether a design has problems worth fixing. It says nothing about how many people a problem affects. If one participant in five struggles with a task, the true proportion of users who would struggle could be anywhere from under 1% to more than 70%, and a test of that size cannot tell which. Nor can a test with a handful of users show which of two competing designs is better when both have a similar number of problems, or how large the benefit of a change would be. These are precisely the questions that decision-makers ask, particularly for systems that will be widely deployed, and answering them requires a different kind of evidence.
Even the original research does not support five as a universal number. Nielsen and Landauer’s own cost-benefit estimates put the optimal number of test users at between 7 and 20, depending on the size of the project, and for a medium-sized example they estimated that 16 evaluations would be worth their cost. Later research also found that results depend heavily on which five people happen to be chosen. In a 2003 study by Laura Faulkner, some randomly selected groups of five participants found 99% of the usability problems in a system, while others found only 55%. The five-user rule is an average drawn from a particular set of studies, not a guarantee for any given test, and treating it as one creates a false sense of certainty.
How much testing is worthwhile depends on what a usability problem costs, and this is where the stakes for government become clear. Nielsen and Landauer estimated the benefit of finding a single usability problem at around $1,000 for a small system used by few people, $15,000 for a system in extensive use, and $200,000 for a system used intensively by very large numbers of people, noting that the figure could reach into the millions, all in 1993 dollars. For a small internal tool, a few test users may well be all the testing that can be justified. For a whole-of-government platform used by millions of people, a single problem left unfound can generate thousands of calls to contact centres, errors that staff must correct, and people who give up on services they are entitled to use. The amount of testing should therefore be set by the cost of what it might miss, not by a rule of thumb.
Yet this trade-off is rarely weighed fairly, because the costs and benefits fall in different places and at different times. The cost of testing is real and immediate, and it comes out of the budget of the project delivering the change. The cost of a usability problem arrives later, and usually lands somewhere else: on the contact centre that takes the calls, the operational area that corrects the errors, the team that inherits the system, and the people who struggle to use it. A project judged on delivering to time and budget has every incentive to test as little as possible, even when the organisation as a whole would save many times the cost. In this situation, a small sample is not a methodological choice but a budgetary one, presented as if the evidence it produces were sufficient.
Questions about how many and how much call for quantitative methods, and these are well established. The number of participants needed depends on two choices: how precise the answer needs to be, known as the margin of error, and how confident we want to be that the true value falls within it, known as the confidence level. To estimate the proportion of users who can complete a task to within 20 percentage points, with 95% confidence, around 24 participants are needed, and a tighter margin requires more. Perhaps counterintuitively, the total number of users barely matters once it runs into the thousands, so the same 24 participants serve as well for a system with a million users as for one with 50,000. This means a study can be budgeted from the precision a decision requires, rather than from habit or guesswork.
Consider a government agency with a platform used by more than a million people. A team believes that redesigning the onboarding process would improve the experience, but implementation will be expensive, and the agency has already conducted its own testing and believes its current design is a good one. A small test of the prototype cannot win that argument, because it can show that problems exist but not that the new design is meaningfully better. Instead, the team could recruit 48 participants, randomly assign half to the existing system and half to the prototype, and measure task completion and satisfaction for each group. If the average rating for onboarding rose from 3.40 to 4.69 on a five-point scale, and statistical testing showed that a difference of that size was very unlikely to have arisen by chance, the team could state how large the improvement was and how confident they were in it. That is a claim an executive can weigh against the cost of building the change.
Matching the method to the question matters because the consequences of getting it wrong fall on real decisions and real people. When small tests are used to make large claims, organisations will continue to fund changes that make little difference, and to reject changes that would have helped, with no way of telling the two apart. And as long as the cost of usability problems is counted somewhere other than where testing is paid for, small tests will keep looking cheaper than they really are. Research itself is weakened as well, since findings that cannot survive scrutiny lose their place at the table, leaving opinion and seniority to decide. For government services in particular, the price of a poor decision is paid by the people using them, often those with the least capacity to work around a design that does not suit them. If user research is to inform how public money is spent, practitioners need to be clear about what each method can and cannot show, and to use the right one for the question being asked.