Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

For the past decade, testing the usability of print software manuals has become a mature area of study, characterized by a wide range of qualitative and quantitative methods. Some of the most common methods include field observations, surveys, interviews, protocol analyses, focus groups, iterative testing, and quasi-experimental lab simulations [1]. Such diverse approaches to usability testing offer an opportunity for complementary inquiries and analyses. For example, findings from focus groups can provide key questions for experimental researchers to pursue in greater depth and with greater possibility for generalizability. Essentially, this complementary approach envisions an interaction between the academy, with its propensity toward pure, experimental research, and industry, with its more applied approaches for alpha and beta testing. Patricia Wright, a specialist in usability studies, has long argued that integrating pure and applied research is the best means for expanding our knowledge about effective document design [2; 3]. Such integration reveals both the immediately applicable aspects of effective manuals and the more theoretical boundaries in textual features that make a difference for general types of tasks, readers, and contexts of use. In order to realize the potential of conducting a conversation between pure and applied research, documentation researchers and practitioners must clearly understand the limitations that exist in the conclusions that investigators derive from specific methods of inquiry. In this article, I look solely at experimental usability tests that rely on quantitative methods of analysis. I analyze the ways in which the research designs and questions of the past ten years of experimental studies affect the strength of cumulative conclusions and the confidence we can have in those conclusions. My purpose is not to give preference to experimental research as the most important approach to usability testing. Far from it. Rather my critical review has two purposes: (1) to facilitate the dialogue between academic and industrial researchers by identifying the limits of current experimental findings; and (2) to propose research agendas and designs for future experimental usability tests that can strengthen the conclusions that such researchers offer for practical consideration. My evaluation of ten years of experimental usability studies shows that many of the conclusions of these studies are not strong enough to serve as valid, generalizable, and replicable foundations for subsequent research, be it pure or applied. These conclusions can be strengthened by designing studies that pay more attention to the sequencing and integration of related investigations and that institute better controls for sample selection, size, and composition. This article discusses my overall findings, the details of which I will develop more fully in my presentation.

REFERENCES

• 1: See the special issue on Usability Testing, IEEE Transactions on Professional Communication PC-32 (December 1989) pp. 205-309
• 2: Patricia Wright, "Can Research Assist Technical Comunicafion?" Proceedings of the 35th ITCC (1989) RT3-6
• 3: Patricia Wright, "Feeding the Informtion ~ot~r~. qllcrcr~et~r~e ~r~r T~t~m-~t~rr D!lr~ o~rl Applied Research in Language Comprehension," Instructional Science 7 (1978) pp. 249-342
• 4: David Kieras and Susan Bovair, !k~ Acauisition of Procedures From Text: A Proctuction-System An01y.sis of Transfer of Trainine, Technical Report No. 16, Office of Naval Research, 1985
• 5: Gregory Krohn, "Flowcharts Used for Procedural Instructions," Human Factors 25 (1983) pp. 573-581.
• 6: D. J. Foss , P. L. Smith-Kerker , M. B. Rosson, On comprehending a computer manual: analysis of variables affecting performance, International Journal of Man-Machine Studies, v.26 n.3, p.277-300, March 1987 [doi>10.1016/S0020-7373(87)80064-0]
• 7: Richard Mayer and Bruce Bromage, "Different Recall Protocols for Technical Texts Due to Advance Organizers," Journal Qf Educational Psychology 22 (1980) pp. 209- 225
• 8: Peter Dixon, "Plans and Written Directions for Complex Tasks," Journal of Verbal Learning and V~rbal Behavior 2 (1982) pp. 70- 84
• 9: Peter Dixon, "The Processing of Organizational and Component Step Information in Written Directions," Journal of Memory_ and Language 26 (1987) pp. 24-35
• 10: John Carroll, Penny Smith-Kerker, Jim Ford, and Sandra Mazur~Rimetz, "The Minimal Manual," Human-Comnuter Interaction 3 (1987-88) pp. 123-153.
• 11: Esther Odescalchi, "Documentation is the Key to User Success," !._E.E.E. Transactions on Professional Common. icafion Vol. PC-29 (19~0) pp. lO-1~
• 12: Robert Holt, Deborah Boehm-Davis, and Alan Schultz, "Multileveled Structured Documentation," Human Factors 31 (1989) pp. 215-228.
• 13: Jaap Vermeulen, Effects of functionally or topographically presented process schemes on operator performance, Human Factors, v.29 n.4, p.383-394, Aug. 1987
• 14: John Carroll, Robert Mack, Clayton Lewis, Nancy Grischkowsky, and Scott Robertson, "Exploring Exploring a Wordprocessor," Human-ComputerInteraction 1 (1985) pp. 283-307.
• 15: Richard Light, "Six Evaluation Issues That Synthesis Can Resolve Better Than Single Studies, in W. H. Yeaton and P. M. Wortman (eds.) New Directions for Proeram Evaluation, (San Francisco: Jossey Bass, 1984)
• 16: Davida Chamey and Lynne Reder, "Designing Interactive Tutorials for Computer Users," Human-Computer Interaction 2 (1986) pp. 287~317
• 17: Piraye Bayman and Richard Mayer, "Instructional Manipulation of Users' Mental Models for Electronic Calculators," International Journal of Man-Machine Studies 20 (1984) pp. 189-199.
• 18: David Schell, "Testing Online and Print User Documentation," I.E.E.E. Transactions on Professional Communication Vol. PC-29 (1986) pp. 87-92.
• 19: Davida Charney, Lynne Reder, and Gail Wells, "Studies of Elaboration in Instructional Texts'" in Stephen Doheny-Farina (ed.) Effective Documentation: What We Have Learned From Research (Cambridge, MA: MIT Press, 1988)
• 20: Frank G. Halasz , Thomas P. Moran, Mental models and problem solving in using a calculator, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, p.212-216, December 12-15, 1983, Boston, Massachusetts, United States [doi>10.1145/800045.801613]
• 21: C Katzeff, Dealing with a database query language in a new situation, International Journal of Man-Machine Studies, v.25 n.1, p.1-17, July 1986 [doi>10.1016/S0020-7373(86)80030-X]
• 22: A L Kamouri , J Kamouri , K H Smith, Training by exploration: facilitating the transfer of procedural knowledge through analogical reasoning, International Journal of Man-Machine Studies, v.24 n.2, p.171-192, Feb. 1986 [doi>10.1016/S0020-7373(86)80047-5]
• 22: John M. Carroll , Caroline Carrithers, Training wheels in a user interface, Communications of the ACM, v.27 n.8, p.800-806, Aug 1984 [doi>10.1145/358198.358218]
• 24: Piraye Bayman and Richard Mayer, "Instructional Manipulation of Users' Mental Models for Electronic Calculators," IntemationalJoumalof Man-Machine Studies 20 (1984) pp. 189-199.
• 25: John M. Carroll , Caroline Carrithers, Training wheels in a user interface, Communications of the ACM, v.27 n.8, p.800-806, Aug 1984 [doi>10.1145/358198.358218]
• 26: John M. Carroll and Caroline Carrithers, "Blocking Learner Error States in a Training- Wheels System," Human Factori 26 (1984) 377-389
• 27: John Carroll, Robert Mack, Clayton Lewis, Nancy Grischkowsky, and Scott Robertson, "Exploring Exploring a Wordprocessor," Human-Computer Interactiog 1 (1985) pp. 283-307.
• 28: John Carroll, Penny S mith-Kerker, Jim Ford, and Sandra Mazur-Rimetz, "The Minimal Manual," Human-Comouter Interaction 3 (1987-88), pp. 123-153.
• 29: Davida Charney and Lynne Reder, "Designing Interactive Tutorials for Computer Users," Human-Computer Interaction 2 (1986) pp. 287-317
• 30: Davida Charney and Lynne Reder, "Initial Skill Learning: An Analysis of How Elaborations Facilitate the Three Components," in Peter Morris (ed.) Modelling Co~ition, (NY: john Wiley and Sons, 1987)
• 31: Davida Chamey, Lynne Reder, and Gall Wells, "Studies of Elaboration in Instructional Texts'" in Stephen Doheny-Farina (ed.) Effective Documentation: What We Have ~amed From Research (Cambridge, MA: MIT Press, 1988)
• 32: Peter Dixon, "Plans and Written Directions for Complex Tasks," J.o.uma_l of Verbal Learning and Verbal Behavior 21 (1982) 70-84
• 33: Peter Dixon, "The Processing of Organizational and Component Step Information in Written Directions," Journal of Memory_ an dLan~uage 26 (1987) 24-35
• 34: Thomas Duffy, Thomas Curran, and Del Sass, "Document Design for Technical Job Tasks' An Evaluation," Human Factors 25 (1983) 143-160
• 35: D. J. Foss , P. L. Smith-Kerker , M. B. Rosson, On comprehending a computer manual: analysis of variables affecting performance, International Journal of Man-Machine Studies, v.26 n.3, p.277-300, March 1987 [doi>10.1016/S0020-7373(87)80064-0]
• 36: Frank G. Halasz , Thomas P. Moran, Mental models and problem solving in using a calculator, Proceedings of the SIGCHI conference on Human Factors in Computing Systems, p.212-216, December 12-15, 1983, Boston, Massachusetts, United States [doi>10.1145/800045.801613]
• 37: Robert Holt, Deborah Boehm-Davis, and Alan Schultz, "Multileveled Structured Documentation," Human Factors 31 (1989), pp. 215-228.
• 38: A L Kamouri , J Kamouri , K H Smith, Training by exploration: facilitating the transfer of procedural knowledge through analogical reasoning, International Journal of Man-Machine Studies, v.24 n.2, p.171-192, Feb. 1986 [doi>10.1016/S0020-7373(86)80047-5]
• 39: C Katzeff, Dealing with a database query language in a new situation, International Journal of Man-Machine Studies, v.25 n.1, p.1-17, July 1986 [doi>10.1016/S0020-7373(86)80030-X]
• 40: David Kieras, The Role of Prior Knowledge in Operatine EQuir~ment FromWritten Instructions, Technical Report No. 19, Office of Naval Research, 1985
• 41: David Kieras and Susan Bovair, Acquisition of Procedures From Text; __A Production-System Analysis of Transfer of Training, Technical Report No. 16, Office of Naval Research, 1985
• 42: Gregory Krohn, "Flowcharts Used for Procedural Instructions," Human Factors 25 (1983), pp. 573-581.
• 43: Richard Mayer and Bruce Bromage, "Different Recall Protocols for Technical Texts Due to Advance Organizers," Journal of Educational Psychology 22 (1980) 209-225
• 44: Esther Odescalchi, "Documentation is the Key to User Success," I.E.E.E. Transactions on Professional Communication Vol. PC-29 (1986), pp. 16-18
• 45: Lynne Reder, Davida Charney, and Kim Morgan, "The Role of Elaborations in Learning a Skill From an Instructional Text," Memory and Cognition 14 (2986) 64-78
• 46: Mark Scerbo, Arthur Fisk, and Joseph Limanowski, "Toward an Understanding of Documentation Usage: Referencing and Problem-Solving Strategies," in R. E. Eberts and C. G. Eberts (eds.) Trends in Ergonomics/Human Factors II (Amsterdam: Noah-Holland, 1985)
• 47: David Schell, "Testing Online and Print User Documentation," I.E.E.E. Transactions on Profe~ional Communication Vol. PC-29 (1986) pp. 87-92.
• 48: Jaap Vermeulen, Effects of functionally or topographically presented process schemes on operator performance, Human Factors, v.29 n.4, p.383-394, Aug. 1987

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.7 Distribution, Maintenance, and Enhancement
          Subjects: Documentation

Additional Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

General Terms:
Design, Documentation, Human Factors