Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality

ABSTRACT

We present Opportunistic Controls, a class of user interaction techniques for augmented reality (AR) applications that support gesturing on, and receiving feedback from, otherwise unused affordances already present in the domain environment. Opportunistic Controls leverage characteristics of these affordances to provide passive haptics that ease gesture input, simplify gesture recognition, and provide tangible feedback to the user. 3D widgets are tightly coupled with affordances to provide visual feedback and hints about the functionality of the control. For example, a set of buttons is mapped to existing tactile features on domain objects. We describe examples of Opportunistic Controls that we have designed and implemented using optical marker tracking, combined with appearance-based gesture recognition. We present the results of a user study in which participants performed a simulated maintenance inspection of an aircraft engine using a set of virtual buttons implemented both as Opportunistic Controls and using simpler passive haptics. Opportunistic Controls allowed participants to complete their tasks significantly faster and were preferred over the baseline technique.

REFERENCES

• 1: Da Pam 738--751, 1992. Functional Users Manual for The Army Maintenance Management System - Aviation (TAMMS-A). Washington D.C: U.S. Army.
• 2: Gabor Blasko , Steven Feiner, An Interaction System for Watch Computers Using Tactile Guidance and Bidirectional Segmented Strokes, Proceedings of the Eighth International Symposium on Wearable Computers, p.120-123, October 31-November 03, 2004 [doi>10.1109/ISWC.2004.6]
• 3: Bleser, G. and Stricker, D. 2008. Advanced tracking through efficient image processing and visual-inertial sensor fusion. Proc. IEEE Virtual Reality Conf. (VR '08), 137--144.
• 4: Frederick P. Brooks, Jr. , Ming Ouh-Young , James J. Batter , P. Jerome Kilpatrick, Project GROPEHaptic displays for scientific visualization, Proceedings of the 17th annual conference on Computer graphics and interactive techniques, p.177-185, September 1990, Dallas, TX, USA
• 5: William Buxton , Ralph Hill , Peter Rowley, Issues and techniques in touch-sensitive tablet input, Proceedings of the 12th annual conference on Computer graphics and interactive techniques, p.215-224, July 1985
• 6: Brookshire D. Conner , Scott S. Snibbe , Kenneth P. Herndon , Daniel C. Robbins , Robert C. Zeleznik , Andries van Dam, Three-dimensional widgets, Proceedings of the 1992 symposium on Interactive 3D graphics, p.183-188, June 1992, Cambridge, Massachusetts, United States [doi>10.1145/147156.147199]
• 7: Mark Fiala, ARTag, a Fiducial Marker System Using Digital Techniques, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.590-596, June 20-26, 2005 [doi>10.1109/CVPR.2005.74]
• 8: Kenneth P. Fishkin, A taxonomy for and analysis of tangible interfaces, Personal and Ubiquitous Computing, v.8 n.5, p.347-358, September 2004 [doi>10.1007/s00779-004-0297-4]
• 9: Gibson, J. 1986. The Ecological Approach to Visual Perception. Hillsdale, N.J.: Lawrence Erlbaum Associates.
• 10: Ken Hinckley , Randy Pausch , John C. Goble , Neal F. Kassell, Passive real-world interface props for neurosurgical visualization, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.452-458, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191821]
• 11: Brent E Insko , Michael J Meehan , Mary C Whitton , Frederick P Brooks, Jr., Passive Haptics Significantly Enhances Virtual Environments, University of North Carolina at Chapel Hill, Chapel Hill, NC, 2001
• 12: Hiroshi Ishii , Brygg Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258715]
• 13: R. Kjeldsen , J. Kender, Finding skin in color images, Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition (FG '96), p.312, October 14-16, 1996
• 14: Klein, G. and Murray, D. 2007. Parallel tracking and mapping for small AR workspaces. Proc. International Symp. on Mixed and Augmented Reality (ISMAR'07).
• 15: Robert W. Lindeman , John L. Sibert , James K. Hahn, Hand-Held Windows: Towards Effective 2D Interaction in Immersive Virtual Environments, Proceedings of the IEEE Virtual Reality, p.205, March 13-17, 1999
• 16: Roderick Murray-Smith , John Williamson , Stephen Hughes , Torben Quaade, Stane: synthesized surfaces for tactile input, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357257]
• 17: Norman, D. 1988. The Psychology of Everyday Things. New York: Basic Books.
• 18: Helena Roeber , John Bacus , Carlo Tomasi, Typing in thin air: the canesta projection keyboard - a new method of interaction with electronic devices, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.765944]
• 19: 3DV Systems, 2008. http://www.3dvsystems.com
• 20: Szalavari, Z. and Gervautz, M. 1997. The personal interaction panel - a two-handed interface for augmented reality Computer Graphics Forum 16, 3, 335--346.
• 21: Carlo Tomasi , Abbas Rafii , Ilhami Torunoglu, Full-size projection keyboard for handheld devices, Communications of the ACM, v.46 n.7, July 2003 [doi>10.1145/792704.792732]
• 22: D. Weimer , S. K. Ganapathy, A synthetic visual environment with hand gesturing and voice input, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.235-240, March 1989

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Artificial, augmented, and virtual realities
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation)

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS
      I.3.6 Methodology and Techniques
          Subjects: Interaction techniques

General Terms:
Human Factors

Keywords:
3D interaction, augmented reality, selection metaphor, tangible user interfaces