Evaluating the usability of an evolving collaborative product —changes in user type, tasks and evaluation methods over time

ABSTRACT

The first users of a new technology are often engineers and enthusiasts. The functionality and interface that they find acceptable may be very different than the requirements of a more mainstream audience. This poses challenges for usability engineers in both defining user groups and then evaluating a product against usability goals, when both users and goals are changing as the technology matures. Usability evaluation methods for collaborative applications must evolve and iterate at least as fast as the products themselves. This paper describes the changes in approach taken by usability engineers between Version 1 and Version 3 of the Microsoft NetMeeting product.

REFERENCES

• 1: Chapanis, A., Ochsman, R., Parrish, R., & Weeks, G. (1972). Studies in interactive communication: I. The effects of four communication modes on the behavior of teams during cooperative problem solving. Human Factors, 14, pp. 487-509.
• 2: Paul Dourish , Sara Bly, Portholes: supporting awareness in a distributed work group, Proceedings of the SIGCHI conference on Human factors in computing systems, p.541-547, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142982]
• 3: Donald R. Gentner , Jonathan Grudin, Design Models for Computer-Human Interfaces, Computer, v.29 n.6, p.28-35, June 1996 [doi>10.1109/2.507629]
• 4: Rebecca E. Grinter, Recomposition: putting it all back together again, Proceedings of the 1998 ACM conference on Computer supported cooperative work, p.393-402, November 14-18, 1998, Seattle, Washington, United States [doi>10.1145/289444.289514]
• 5: Jonathan Grudin, Interactive Systems: Bridging the Gaps Between Developers and Users, Computer, v.24 n.4, p.59-69, April 1991 [doi>10.1109/2.76263]
• 6: Hiroshi Ishii , Minoru Kobayashi , Kazuho Arita, Iterative design of seamless collaboration media, Communications of the ACM, v.37 n.8, p.83-97, Aug. 1994 [doi>10.1145/179606.179687]
• 7: Gloria Mark , Jonathan Grudin , Steven E. Poltrock, Meeting at the desktop: an empirical study of virtually collocated teams, Proceedings of the Sixth European conference on Computer supported cooperative work, p.159-178, August 1999, Copenghagen, Denmark
• 8: Deborah J. Mayhew, The usability engineering lifecycle: a practitioner's handbook for user interface design, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 9: Judith S. Olson , Gary M. Olson , David K. Meader, What mix of video and audio is useful for small groups doing remote real-time design work?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.362-368, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223951]
• 10: Palen, L. (1999). Convergent Perspectives on Groupware Evaluation & Design. CHI 99 Proceedings, May 15-20, 1999, Pittsburgh, PA USA
• 11: Abigail J. Sellen, Speech patterns in video-mediated conversations, Proceedings of the SIGCHI conference on Human factors in computing systems, p.49-59, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142756]
• 12: John C. Tang , Monica Rua, Montage: providing teleproximity for distributed groups, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.37-43, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191688]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Design, Theory, Verification

Keywords:
NetMeeting, application sharing, collaborative tools, design consideration, development process, usability evaluation