Obstacles to usability evaluation in practice: a survey of software development organizations

ABSTRACT

This paper reports from a combined questionnaire survey and interview study of obstacles for deploying usability evaluation in software development organizations. It was conducted in a limited geographical area. The purpose of the questionnaire survey was to determine whether software development organizations in that area were evaluating the usability of their software and to identify key obstacles. It revealed that 29 of 39 software development organizations conducted some form of usability evaluation. The purpose of the interview study was to gain more insight into the obstacles that were expressed. It involved 10 of the 39 software development organizations. Our results show, that the understanding of usability evaluation is a major obstacle. Furthermore, the two most significant obstacles were resource demands and the mindset of developers. These obstacles were not only an obstacle for more organizations to deploy usability evaluation, but also a concern for the software development organizations, that had deployed usability evaluations in their development process.

REFERENCES

• 1: Beu, A., Honold, P., and Yuan, X. 2000 How to Build Up an Infrastructure for Intercultural Usability Engineering. International Journal of Human-Computer Interaction 12, 3--4, 347--358.
• 2: Susanne Bødker , Jacob Buur, The design collaboratorium: a place for usability design, ACM Transactions on Computer-Human Interaction (TOCHI), v.9 n.2, p.152-169, June 2002 [doi>10.1145/513665.513670]
• 3: Boivie, I., Aaborg, C., Persson, J., and Löfberg, M. 2003 Why usability gets lost or usability in in-house software development. Interacting with Computers 15, 4, 623--639.
• 4: Jan Gulliksen , Inger Boivie , Bengt Göransson, Usability professionals-current practices and future development, Interacting with Computers, v.18 n.4, p.568-600, July, 2006 [doi>10.1016/j.intcom.2005.10.005]
• 5: Inger Boivie , Jan Gulliksen , Bengt Göransson, The lonesome cowboy: A study of the usability designer role in systems development, Interacting with Computers, v.18 n.4, p.601-634, July, 2006 [doi>10.1016/j.intcom.2005.10.003]
• 6: Åsa Cajander , Jan Gulliksen , Inger Boivie, Management perspectives on usability in a public authority: a case study, Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles, p.38-47, October 14-18, 2006, Oslo, Norway [doi>10.1145/1182475.1182480]
• 7: Carter, J. 1999 Incorporating standards and guidelines in an approach that balances usability concerns for developers and end users. Interacting with Computers 12, 2, 179--206.
• 8: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, L. Erlbaum Associates Inc., Hillsdale, NJ, 1983
• 9: Dillon, A., Sweeney, M., and Maguire, M. 1993 A survey of usability engineering within the European I.T. Industry Current practice and needs. Proceedings of CHI 93. ACM Press.
• 10: Susan M. Dray , Clare-Marie Karat, Human factors cost justification for an internal development project, Cost-justifying usability, Academic Press, Inc., Orlando, FL, 1994
• 11: Gulliksen, J., Göransson, B., Boivie, I., Blomkvist, S., Persson, J., and Cajander, Aa. 2003 Key principles for user-centred systems design. Behaviour & Information Technology 22, 6, 397--409.
• 12: Granollers, T. 2003 User Centred Design Process Model. Integration of Usability Engineering and Software Engineering. Proceedings of Interact 2003.
• 13: Jonathan Grudin, Obstacles to user involvement in software product development, with implications for CSCW, International Journal of Man-Machine Studies, v.34 n.3, p.435-452, March 1991 [doi>10.1016/0020-7373(91)90029-7]
• 14: Jonathan Grudin, Systematic sources of suboptimal interface design in large product development organizations, Human-Computer Interaction, v.6 n.2, p.147-196, June 1991 [doi>10.1207/s15327051hci0602_3]
• 15: Jonathan Grudin, Interactive Systems: Bridging the Gaps Between Developers and Users, Computer, v.24 n.4, p.59-69, April 1991 [doi>10.1109/2.76263]
• 16: John D. Gould , Clayton Lewis, Designing for usability: key principles and what designers think, Communications of the ACM, v.28 n.3, p.300-311, March 1985 [doi>10.1145/3166.3170]
• 17: Gunther, R., Janis, J., and Butler S. The UCD Decision Matrix: How, When, and Where to Sell User-Centered Design into the Development Cycle. Found at: http://www.ovostudios.com/upa2001/surf.htm last seen April 11th 2007.
• 18: HTS-A's Regionalraad 2006 HTS-A Region Nordjylland The voice of the service sector (in Danish). HTS-A.
• 19: N. Iivari, 'Representing the User' in software development-a cultural analysis of usability work in the product development context, Interacting with Computers, v.18 n.4, p.635-664, July, 2006 [doi>10.1016/j.intcom.2005.10.002]
• 20: Ji, Y. G. and Yun, M. H. 2006 Enhancing the Minority Discipline in the IT Industry: A Survey of Usability and User-Centered Design Practice. International Journal of Human-Computer Interaction 20, 2, 117--134.
• 21: Clare-Marie Karat, A comparison of user interface evaluation methods, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 22: Karat. C.-M. 1997 Cost-justifying usability engineering in the software life cycle. Handbook of Human-Computer Interaction, 767--778.
• 23: Kujala, S. User involvement: a review of the benefits and challenges. Behaviour & Information Technology 22, 1, 1--16.
• 24: Kvale, S. 1998 Interview. An introduction to the qualitative research interview, 2^nd ed. Hans Reitzel.
• 25: Søren Lauesen, Usability Engineering in Industrial Practice, Proceedings of the IFIP TC13 Interantional Conference on Human-Computer Interaction, p.15-22, July 14-18, 1997
• 26: Reed, S. 1992 Who defines usability? You do!. PC//Computing Dec., 220--232.
• 27: Stephanie Rosenbaum , Sarah Bloomer , Dave Rinehart , Janice Rohn , Ken Dye , Judee Humburg , Jakob Nielsen , Dennis Wixon, What makes strategic usability fail?: lessons learned from the field, CHI '99 extended abstracts on Human factors in computing systems, May 15-20, 1999, Pittsburgh, Pennsylvania [doi>10.1145/632716.632773]
• 28: Stephanie Rosenbaum , Janice Anne Rohn , Judee Humburg, A toolkit for strategic usability: results from workshops, panels, and surveys, Proceedings of the SIGCHI conference on Human factors in computing systems, p.337-344, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332454]
• 29: Scapin, D. and Law, E. (2007) Proceedings of Mause International Workshop on Review, Report and Refine Usability Evaluation Methods (R3-UEM). COST 294 MAUSE.
• 30: Smith, A. and Dunckley, L. 2002 Prototype evaluation and redesign: structuring the design space through contextual techniques. Interacting with Computers 14, 6, 821--843.
• 31: Strauss, A. and Corbin, J. 1998 Basics of qualitative research. Techniques and Procedures for Developing Grounded Theory, 2.edition. SAGE Publications.
• 32: Strøm, G. How usability practitioners can get their suggestions implemented in industrial software design. Found at: http://www.georg.dk/usab-imple.pdf last seen April 11th 2007.
• 33: Videnskabsministeriet 2005 A sector that creates value? Focus on the Danish IT sector (in Danish) Ministeriet for Videnskab Teknologi og Udvikling.
• 34: Stephanie Wilson , Mathilde Bekker , Peter Johnson , Hilary Johnson, Helping and hindering user involvement — a tale of everyday design, Proceedings of the SIGCHI conference on Human factors in computing systems, p.178-185, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258699]
• 35: Wynekoop, J. L. and Conger, S. A. 1990 A Review of Computer Aided Software Engineering Research Methods. Proceedings of the IFIP TC8WG8.2 Working Conference on The Information Systems Research Arena of The 90s.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Life cycle

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.9 Management
          Subjects: Software quality assurance (SQA)

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.1 Project and People Management
          Subjects: Systems development

General Terms:
Human Factors, Management

Keywords:
obstacles, software development, usability evaluation