Usability benchmark study of commercially available smart phones: cell phone type platform, PDA type platform and PC type platform

ABSTRACT

The upcoming of smart phones are the result of consumers' preference for high-feature products: manufacturers are lured into integration of a growing number of technologies and features to provide attractive and competitive models. At the same time usability of such complex products becomes an increasing problem. This study aims to provide designers more insight into the consequences of emerging technologies on the usability of smart phones with different platforms. We conducted a usability benchmark study and tested 4 different features on 3 smart phones with 3 different platforms (Nokia E6li (Symbian S60, cell phone type platform), HTC S710 (Windows Mobile 6, PC type platform), Palm Treo680 (Palm OS, PDA type platform)) with in total 43 subjects in a between-subjects design. The results show significant differences in usability of the smart phones for the selected functions. For all platforms, the common design guidelines for usability still apply, but are restricted by the desire to integrate more functionality to create competitive products.

REFERENCES

• 1: Robert St. Amant , Thomas E. Horton , Frank E. Ritter, Model-based evaluation of cell phone menu interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.343-350, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985736]
• 2: Bay, S. and Ziefle, M. Children using cellular phones: The effects of shortcomings in user interface design. Human Factors 47, 1 (2005), 158--168.
• 3: Brown, C. L. and Carpenter, G. S. Why is trivial important? A reasons-based account for the effects of trivial attributes on choice. Journal of Consumer Research 26, (2000), 372--385.
• 4: George Buchanan , Sarah Farrant , Matt Jones , Harold Thimbleby , Gary Marsden , Michael Pazzani, Improving mobile internet usability, Proceedings of the 10th international conference on World Wide Web, p.673-680, May 01-05, 2001, Hong Kong, Hong Kong [doi>10.1145/371920.372181]
• 5: Chae, M. and Kim, J. Do size and structure matter to mobile users? An empirical study of the effects of screen size, information structure, and task complexity on user activities with standard web phones. Behaviour & Information Technology 23, 3 (2004), pp 165--181.
• 6: Andy Cockburn , Carl Gutwin , Saul Greenberg, A predictive model of menu performance, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240723]
• 7: Alan Cooper , Paul Saffo, The Inmates Are Running the Asylum, Macmillan Publishing Co., Inc., Indianapolis, IN, 1999
• 8: Cooper, Robert G. Winning at new products: accelerating the process from idea to launch. 3rd edition. Perseus Publishing, New York, 2001.
• 9: Cordell, V. V. Consumer knowledge measures as predictors in product evaluation. Psychology & Marketing 14, 3 (1997), 241--260.
• 10: Den Ouden, E., Lu, Y., Sonnemans, P. J. M. and Brombacher, A. C. Quality and reliability problems from a consumer's perspective: an increasing problem overlooked by businesses? Quality and Reliability Engineering International 22, 7 (2006), 821--838.
• 11: Joseph S. Dumas , Janice C. Redish, A Practical Guide to Usability Testing, Intellect Books, Exeter, UK, 1999
• 12: Flynn, L. R. and Goldsmith, R. E. A short, reliable measure of subjective knowledge. Journal of Business Research 46, (1999) 57--66.
• 13: Mikael Goldstein , Mikael Anneroth , Robert Book, Usability evaluation of a high-fidelity smart phone prototype: task navigation depth affects effectiveness, Proceedings of the HCI International '99 (the 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Communication, Cooperation, and Application Design-Volume 2, p.38-42, August 22-26, 1999
• 14: Gourville, J. T. Eager sellers and stony buyers - Understanding the psychology of new-product adoption, Harvard Business Review 84, (2006), 98--106.
• 15: Hochheiser, H. and Shneiderman, B. Performance benefits of simultaneous over sequential menus as task complexity increases. International Journal of Human-Computer Interaction 12, 2 (2000), 173--192.
• 16: Paul Holleis , Friederike Otto , Heinrich Hussmann , Albrecht Schmidt, Keystroke-level model for advanced mobile phone interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240851]
• 17: Howell, D. C. Statistical Methods for Psychology, Fifth Edition, Thompson Learning, Pacific Grove, 2002.
• 18: ISO 9241-11:1998 Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) - Part 11: Guidance on Usability, 1998.
• 19: Jeong, S.-H. and Lee, K.-P. The effects of experience with a PC on the usability of a mobile product, In Proc. 6th Asian Design International Conference, 2003.
• 20: Kiljander, H. Evolution and Usability of Mobile Phone Interaction Styles. Thesis for Doctor of Science in Technology, Helsinki University of Technology, December 2004.
• 21: James R. Lewis, Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ, ACM SIGCHI Bulletin, v.23 n.1, p.78-81, Jan. 1991 [doi>10.1145/122672.122692]
• 22: Malykhina, E. Analysis: How smartphone platforms compare. InformationWeek, 20 January 2007. http://www.informationweek.com/story/showArticle.jhtml?a rticleID=196902226 (accessed August 2007).
• 23: Malykhina, E. Smartphone buyer's guide. InformationWeek, 30 October 2006. http://www.informationweek.com/story/showArticle.jhtml?a rticleID=193402812 (accessed August 2007).
• 24: Mohageg, M. F. and Wagner, A. Design Considerations for Information Appliances. In Information Appliances and Beyond. Interaction Design for Consumer Products, ed. Bergman, E. Academic Press, San Diego, 2000, 27--51.
• 25: Donald A. Norman, The invisible computer, MIT Press, Cambridge, MA, 1998
• 26: Nyberg, M., Björk, S., Goldstein, M. and Redström, J. Handheld applications design: Merging information appliances without affecting usability. In Proc. IFIP TC. 13 Conference on Human Computer Interaction. IOP Press (2001), 391--398.
• 27: Overton, D. 'No Fault Found' returns cost the mobile industry $4.5 billion per year. July 2006. http://www.wdsglobal.com/news/whitepapers/20060717/200 60717.asp (accessed August 2007).
• 28: Rust, R. T., Thompson, D. V., Hamilton, R. W. Defeating Feature Fatigue, Harvard Business Review 84, 2 (2006), 98--107.
• 29: Svendsen, B. Mobile computing. In True Visions, The emergence of ambient intelligence, Aarts, E. and Encarnação, J. L. (Eds.), Berlin, Springer-Verlag, 2006.
• 30: Peter Tarasewich, Wireless devices for mobile commerce: user interface design and usability, Mobile commerce: technology, theory, and applications, IGI Publishing, Hershey, PA, 2003
• 31: Thompson, D. V., Hamilton, R. W. and Rust, R. T. Feature fatigue: When product capabilities become too much of a good thing. Journal of Marketing Research 42, 4 (2005), 431--442.
• 32: Scott Weiss, Handheld Usability, John Wiley & Sons, Inc., New York, NY, 2002
• 33: Zhang, D. and Adipat, B. Challenges, methodologies, and issues in the usability testing of mobile applications. International Journal of Human-Computer Interaction 18, 3 (2005), 293--308.
• 34: Pei Zheng , Lionel Ni, Smart Phone and Next Generation Mobile Computing (Morgan Kaufmann Series in Networking (Paperback)), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
• 35: Martina Ziefle , Susanne Bay, How to overcome disorientation in mobile phone menus: a comparison of two different types of navigation aids, Human-Computer Interaction, v.21 n.4, p.393-433, November 2006 [doi>10.1207/s15327051hci2104_2]
• 36: Ziefle, M., Bay, S. and Schwade, A. On keys' meanings and modes: The impact of different key solutions on children's efficiency using a mobile phone. Behaviour & Information Technology 25, 5 (2006), 413--431.
• 37: Ziefle, M. and Bay, S. How older adults meet complexity: aging effects on the usability of different mobile phones. Behaviour & Information Technology 24, 5 (2005), 375--389.
• 38: Ziefle, M., and Bay, S. Mental models of a cellular phone menu. Comparing older and younger novice users. In Proc. MobileCHI 2004, Springer-Verlag (2004), 25--37.
• 39: Ziefle, M. The influence of user expertise and phone complexity on performance, ease of use and learnability of different mobile phones. Behaviour & Information Technology 21, 5 (2002), 303--311

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

General Terms:
Design, Experimentation, Human Factors

Keywords:
design, smart phone, usability evaluation, user interface