Interact, excite, and feel

ABSTRACT

This paper presents a dynamic system approach to the design of multimodal interactive systems. We use an example where we support human behavior in a browsing task, by adapting the dynamics of navigation using speed-dependent automatic zooming (SDAZ), allowing the user to switch smoothly among different modes of control. We show how the user's intention is coupled to the browsing technique via the dynamic model, and how the SDAZ method couples the document structure to audio samples using a model-based sonification. We demonstrate that this approach is well suited to mobile and wearable applications, and audio feedback provides valuable information, supporting intermittent interaction, i.e. allowing movement-based interaction techniques to continue while the user is simultaneously involved with real life tasks.

REFERENCES

• 1: Michel Beaudouin-Lafon, Designing interaction, not interfaces, Proceedings of the working conference on Advanced visual interfaces, May 25-28, 2004, Gallipoli, Italy [doi>10.1145/989863.989865]
• 2: Stephen A. Brewster , Peter C. Wright , Alistair D. N. Edwards, The design and evaluation of an auditory-enhanced scrollbar, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.173-179, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191733]
• 3: T. Büring. Zoomable user interfaces on small screens - Presentation and interaction design for pen-operated mobile devices. PhD thesis, University of Konstanz, Germany, 2007.
• 4: A. Cockburn, J. Looser, and J. Savage. Around the World in Seconds with Speed-Dependent Automatic Zooming. In ACM Conference on Human Factors in Computing Systems, Vancouver, Canada, pages 35--36, 2003.
• 5: A. Cockburn and J. Savage. Comparing Speed-Dependent Automatic Zooming with Traditional Scroll, Pan, and Zoom Methods. In People and Computers XVII: British Computer Society Conference on Human Computer Interaction, Bath, England, pages 87--102, 2003.
• 6: G. Doherty and M. Massink. Continuous Interaction and Human Control. In J. Alty, editor, Proceedings of the XVIII European Annual Conference on Human Decision Making and Manual Control, pages 80--96, Group-D Publications, Loughborough, 1999.
• 7: P. Eslambolchilar. Making Sense of Interaction Using a Model-Based Approach. PhD thesis, Hamilton Institute, National University of Ireland, NUIM, Maynooth, Ireland, October 2006.
• 8: P. Eslambolchilar and R. Murray-Smith. Tilt-based Automatic Zooming and Scaling in mobile devices-a state-space implementation. In S. A. Brewster and M. D. Dunlop, editors, Mobile Human-Computer Interaction-MobileHCI 2004: 6th International Symposium, volume 3160 of Lecture Notes in Computing Science, pages 120--131, Glasgow, Scotland, September 2004. Springer-Verlag.
• 9: P. Eslambolchilar and R. Murray-Smith. Model-Based, Multimodal Interaction in Document Browsing. In 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms, MLMI, volume 4299 of Lecture Notes in Computer Science, pages 1--12, Washington DC., USA, May 2006. Springer Berlin.
• 10: P. Eslambolchilar, R. Murray-Smith, A. Crossan, S. Dalzel-Job, and F. Pollick. Handbook of Research on User Interface Design and Evaluation for Mobile Technology, chapter Model-based Target Sonification in Small Screen Devices: Perception and Action. Idea Group Reference, 2008.
• 11: G. Faconti and M. Massink. Continuous interaction with computers: Issues and Requirements. In C. Stephanidis, editor, Proceedings of Universal Access in HCI, Universal Access in HCI - HCI International 2001, volume 3, pages 301--304, New Orleans, 2001. Lawrence Erlbaum Associates, Inc.
• 12: P. M. Fitts. The Information Capacity of the Human Motor system in Controlling the Amplitude of Movement. In Journal of Experimental Psychology, volume 47, pages 381--391, 1954.
• 13: W. W. Gaver. Auditory Interfaces. Handbook of Human-Computer Interaction, 2nd edition, 1997.
• 14: T. Hermann and A. Hunt. The discipline of interactive sonification. In T. Hermann and A. Hunt, editors, Proceedings of the International workshop on interactive sonification, Bielefeld, Germany, January 2004.
• 15: T. Hermann and H. Ritter. Listen to your data: Model-based sonification for data analysis. In G. E. Lasker, editor, intelligent computing and multimedia systems, pages 189--194, Baden-Baden, Germany, September 1999.
• 16: Ken Hinckley , Jeff Pierce , Mike Sinclair , Eric Horvitz, Sensing techniques for mobile interaction, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.91-100, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354417]
• 17: Takeo Igarashi , Ken Hinckley, Speed-dependent automatic zooming for browsing large documents, Proceedings of the 13th annual ACM symposium on User interface software and technology, p.139-148, November 06-08, 2000, San Diego, California, United States [doi>10.1145/354401.354435]
• 18: R. J. Jagacinski and J. M. Flach. Control Theory for Humans: Quantitative approaches to modeling performance. Lawrence Erlbaum Associates, Inc., Mahwah, New Jersey, 2003.
• 19: Steve Jones , Matt Jones , Gary Marsden , Dynal Patel , Andy Cockburn, An evaluation of integrated zooming and scrolling on small screens, International Journal of Human-Computer Studies, v.63 n.3, p.271-303, September 2005 [doi>10.1016/j.ijhcs.2005.03.005]
• 20: David J. C. MacKay, Information Theory, Inference & Learning Algorithms, Cambridge University Press, New York, NY, 2002
• 21: Malcolm McCullough, Abstracting Craft: The Practiced Digital Hand, MIT Press, Cambridge, MA, 1998
• 22: I. Oakley, J. Ängeslevä, S. Hughes, and S. O'Modhrain. Tilt and feel: Scrolling with vibrotactile display. In Proceedings of Eurohaptics 2004, Munich Germany, 2004.
• 23: Katsuhiko Ogata, Modern Control Engineering, Prentice Hall PTR, Upper Saddle River, NJ, 1990
• 24: Dynal Patel , Gary Marsden , Matt Jones , Steve Jones, Improving photo searching interfaces for small-screen mobile computers, Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, September 12-15, 2006, Helsinki, Finland [doi>10.1145/1152215.1152247]
• 25: D. Patel, G. Marsden, S. Jones, and M. Jones. An evaluation of techniques for browsing photograph collections on small displays. In Mobile Human-Computer Interaction-Mobile HCI 2004, Lecture Notes in Computer Science, pages 271--303, Glasgow, Scotland, September 2004. Springer.
• 26: J. Savage. The calibration and evaluation of speed-dependent automatic zooming interfaces. Master's thesis, Computer Science Department, University of Canterbury, New Zealand, 2004.
• 27: M. B. Tischler. Advances in Aircraft flight Control. Taylor & Francis, 1994.
• 28: A. Wallace. The Calibration and Optimization of Speed-Dependent Automatic Zooming. Honours Report, University of Canterbury, Christchurch, New Zealand, November 2003.
• 29: Andrew Wallace , Joshua Savage , Andy Cockburn, Rapid visual flow: how fast is too fast?, Proceedings of the fifth conference on Australasian user interface, p.117-122, January 01, 2004, Dunedin, New Zealand
• 30: John Williamson , Roderick Murray-Smith, Sonification of Probabilistic Feedback through Granular Synthesis, IEEE MultiMedia, v.12 n.2, p.45-52, April 2005 [doi>10.1109/MMUL.2005.37]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
      H.5.5 Sound and Music Computing

Keywords:
dynamic continuous interaction, mobile devices, mode switching, multimodality, sonification, speed-dependent automatic zooming