A mathematical model of the finding of usability problems

ABSTRACT

For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.

REFERENCES

• 1: Dalal, S.R., and Mallows, C.L. (1988). When should one stop testing software? J. American Statistical Association 83, 403 (September), 872-879.
• 2: Dalai, S.R., and Mallows, C.L. (1990). Some graphical aids for deciding when to stop testing software. IEEE J. Selected Areas in Communication 8, 2 (February), 169- 175.
• 3: Erhan, S. (1975). Introduction to Stochastic Processes. Prentice Hall, Englewood Cliffs, NJ. p. 87.
• 4: Wayne D. Gray , Bonnie E. John , Michael E. Atwood, The precis of Project Ernestine or an overview of a validation of GOMS, Proceedings of the SIGCHI conference on Human factors in computing systems, p.307-312, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142821]
• 5: Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
• 6: Anker Helms Jørgensen, Using the thinking-aloud method in system development, Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.), p.743-750, September 1989, Boston, Massachusetts, United States
• 7: Claire-Marie Karat , Robert Campbell , Tarra Fiegel, Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.397-404, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142873]
• 8: Lewis, C. (1982). Using the 'thinking-aloud' method in cognitive interface design. Research Report RC-9265, IBM T.J. Watson Research Center, Yorktown Heights, NY.
• 9: James R. Lewis , Suzanne C. Henry , Robert L. Mack, Integrated office software benchmarks: A case study, Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction, p.337-343, August 27-31, 1990
• 10: Marilyn M. Mantei , Toby J. Teorey, Cost/benefit analysis for incorporating human factors in the software lifecycle, Communications of the ACM, v.31 n.4, p.428-439, April 1988 [doi>10.1145/42404.42408]
• 11: Rolf Molich , Jakob Nielsen, Improving a human-computer dialogue, Communications of the ACM, v.33 n.3, p.338-348, March 1990 [doi>10.1145/77481.77486]
• 12: Jakob Nielsen, Usability engineering at a discount, Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.), p.394-401, September 1989, Boston, Massachusetts, United States
• 13: IEEE Software Staff, Big Paybacks from 'Discount' Usability Engineering, IEEE Software, v.7 n.3, p.107-108, May 1990
• 14: Jakob Nielsen, Evaluating the thinking-aloud technique for use by computer scientists, Advances in human-computer interaction (vol. 3), Ablex Publishing Corp., Norwood, NJ, 1993
• 15: Jakob Nielsen, Finding usability problems through heuristic evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.373-380, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142834]
• 16: Jakob Nielsen, The Usability Engineering Life Cycle, Computer, v.25 n.3, p.12-22, March 1992 [doi>10.1109/2.121503]
• 17: Nielsen, J. (1993). Estimating the number of subjects needed for a thinking aloud test. Intl. J. Man-Machine Studies in press.
• 18: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 19: Jakob Nielsen, Iterative User-Interface Design, Computer, v.26 n.11, p.32-41, November 1993 [doi>10.1109/2.241424]
• 20: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 21: Jakob Nielsen , Rolf Molich, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.249-256, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97281]
• 22: Virzi, R.A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society 34th Annual Meeting (Orlando, FL, 8-12 October), 291-294.
• 23: Robert A. Virzi, Refining the test phase of usability evaluation: how many subjects is enough?, Human Factors, v.34 n.4, p.457-468, Aug. 1992
• 24: Peter C. Wright , Andrew F. Monk, A cost-effective evaluation method for use by designers, International Journal of Man-Machine Studies, v.35 n.6, p.891-912, Dec. 1991 [doi>10.1016/S0020-7373(05)80167-1]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

General Terms:
Design, Human Factors, Performance

Keywords:
Poisson models, cost-benefit analysis, heuristic evaluation, iterative design, usability engineering, usability problems, user testing