Estimating Sample Size for Usability Testing

Authors

  • Alex Cazañas University of Queensland, Brisbane
  • Andre de San Miguel University of Queensland, Brisbane
  • Esther Parra Escuela Politécnica Naciona

DOI:

https://doi.org/10.29019/enfoqueute.v8n1.126

Keywords:

usability testing, problem discovery, sample size

Abstract

One strategy used to assure that an interface meets user requirements is to conduct usability testing. When conducting such testing one of the unknowns is sample size. Since extensive testing is costly, minimizing the number of participants can contribute greatly to successful resource management of a project. Even though a significant number of models have been proposed to estimate sample size in usability testing, there is still not consensus on the optimal size. Several studies claim that 3 to 5 users suffice to uncover 80% of problems in a software interface. However, many other studies challenge this assertion. This study analyzed data collected from the user testing of a web application to verify the rule of thumb, commonly known as the “magic number 5”. The outcomes of the analysis showed that the 5-user rule significantly underestimates the required sample size to achieve reasonable levels of problem detection.

Metrics

Downloads

Download data is not yet available.

References

Bevan, N., Barnum, C., Cockton, G., Nielsen, J., Spool, J., & Wixon , D. (2003). The "magic number 5": is it enough for web testing? In ACM (Ed.), CHI '03 Extended Abstracts on Human Factors in Computing Systems (CHI EA '03), (pp. 698-699). Ft. Lauderdale, Florida, USA.
Caulton, D. (2001). Relaxing the homogeneity assumption in usability testing. Behaviour & Information Technology, 20(1), 1-7.
Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379-383.
Hertzum, M., & Jacobsen, N. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 13(4), 421-443.
Hwang, W., & Salvendy , G. (2010). Number of people required for usability evaluation: the 10±2 rule. Commun. ACM, 53(5), 130-133.
Lewis. (1994, June). Sample sizes for usability studies: Additional considerations. Human Factors: The Journal of the Human Factors and Ergonomics Society, 36(2), 368-378.
Lewis. (2000). Overestimation of p in problem discovery usability studies: How serious is the problem. Tech. Rep.
Lewis. (2000). Validation of Monte Carlo estimation of problem discovery. Raleigh, NC: International Business Machines Corp.
Lewis. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13(4), 445-479.
Lindgaard, G., & Chattratichart , J. (2007). Usability testing: what have we overlooked? In ACM (Ed.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (pp. 1415-1424). San Jose, California, USA.
Nielsen, J., & Landauer, T. (1993). A Mathematical Model of the Finding of Usability Problems. In ACM (Ed.), Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems, (pp. 206-213). Amsterdam, The Netherlands.
Spool, J., & Schroeder , W. (2001). Testing web sites: five users is nowhere near enough. In ACM (Ed.), CHI '01 Extended Abstracts on Human Factors in Computing Systems, (pp. 285-286). Seattle, Washington.
Turner, C., Lewis, J., & Nielsen, J. (2006). Determining usability test sample size. International encyclopedia of ergonomics and human factors, 3(2), 3084-3088.
Virzi, R. (1992, August). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors: The Journal of the Human Factors and Ergonomics Society, 34(4), 457-471.
Woolrych, A., & Cockton, G. (2001). Why and when five test users aren't enough. In C. Editions (Ed.), Proceedings of IHM-HCI 2001 conference, 2, pp. 105-108. Toulouse, FR.

Published

2017-02-24

How to Cite

Cazañas, A., de San Miguel, A., & Parra, E. (2017). Estimating Sample Size for Usability Testing. Enfoque UTE, 8(1), pp. 172-185. https://doi.org/10.29019/enfoqueute.v8n1.126

Issue

Vol. 8 No. 1 (2017): INCISCOS 2016, Special Edition

Section

Miscellaneous

The articles and research published by the UTE University are carried out under the Open Access regime in electronic format. By submitting an article to any of the scientific journals of the UTE University, the author or authors accept these conditions.

The UTE applies the Creative Commons Attribution (CC-BY) license to articles in its scientific journals. Under this open access license, as an author you agree that anyone may reuse your article in whole or in part for any purpose, free of charge, including commercial purposes. Anyone can copy, distribute or reuse the content as long as the author and original source are correctly cited. This facilitates freedom of reuse and also ensures that content can be extracted without barriers for research needs.

 

This work is licensed under a Creative Commons Attribution 3.0 International (CC BY 3.0).

 

In addition, the journal Enfoque UTE guarantees and declares that authors always retain all copyrights to the original published works without restrictions [© The Author(s)]. Acknowledgment (BY): Any exploitation of the work is allowed, including a commercial purpose, as well as the creation of derivative works, the distribution of which is also allowed without any restriction.