An architecture for transforming graphical interfaces

ABSTRACT

While graphical user interfaces have gained much popularity in recent years, there are situations when the need to use existing applications in a nonvisual modality is clear. Examples of such situations include the use of applications on hand-held devices with limited screen space (or even no screen space, as in the case of telephones), or users with visual impairments.We have developed an architecture capable of transforming the graphical interfaces of existing applications into powerful intuitive nonvisual interfaces. Our system, called Mercator, provides new input and output techniques for working in the nonvisual domain. Navigation is accomplished by traversing a hierarchical tree representation of the interface structure. Output is primarily auditory, although other output modalities (such as tactile) can be used as well. The mouse, an inherently visually-oriented device, is replaced by keyboard and voice interaction.Our system is currently in its third major revision. We have gained insight into both the nonvisual interfaces presented by our system and the architecture necessary to construct such interfaces. This architecture uses several novel techniques to efficiently and flexibly map graphical interfaces into new modalities.

REFERENCES

• BBV90: L.H. Boy& W.L. Boy& and G.C. Vanderheiden. The graphical user interface: Crisis, danger and opportunity. Journal of Visual Impairment and Blindness. pages 496-502, December 1990.
• Bur92: David Burgess. Low Cost Sound Spatilization. In UIST '92: The Fifth Annual Symposium on User Interface Software and Technology and Technology, November 1992.
• Bux86: W. Buxton , R. Foulds , M. Rosen , L. Scadden , F. Shein, Human interface design and the handicapped user, Proceedings of the SIGCHI conference on Human factors in computing systems, p.291-297, April 13-17, 1986, Boston, Massachusetts, United States
• Gav89: William W. Gaver. The sonicfinder: An interface that uses auditory icons. Human Computer Interaction, 4:67-94, 1989.
• LC91: Michael Cohen , Lester F. Ludwig, Multidimensional audio window management, International Journal of Man-Machine Studies, v.34 n.3, p.319-336, March 1991 [doi>10.1016/0020-7373(91)90023-Z]
• My94: Mynatt, E.D., "Mapping GUIs to Auditory Interfaces, In Kramer G, (ed), Auditory Display: The Proceedings of ICAD '92. SFI Studies in the Sciences of Complexity Proc. Vol. XVIii, Addison-Wesley, April 1994.
• ME92: Elizabeth D. Mynatt , W. Keith Edwards, Mapping GUIs to auditory interfaces, Proceedings of the 5th annual ACM symposium on User interface software and technology, p.61-70, November 15-18, 1992, Monteray, California, United States [doi>10.1145/142621.142629]
• MW94: Elizabeth D. Mynatt , Gerhard Weber, Nonvisual presentation of graphical user interfaces: contrasting two approaches, Conference companion on Human factors in computing systems, p.211, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/259963.260338]
• Ous90: J.K. Ousterhout. "TCL: An Embeddable Command Language," in the Proceedings of the 1990 Winter USENIX Conference, pp. 133- 146.
• Pet91: Chris D. Peterson. Editres-a graphical resource editor for x toolkit applications. In Conference Proceedings, Fifth Annual X Technical Conference, Boston, Massachusetts, January, 1991.
• Sch87: Robert W. Scheitter. X window system protocol specification, version 11. Massachusetts Institute of Technology, Cambridge, Massachusetts, and Digital Equipment Corporation, Maynard, Massachusetts, 1987.
• Ste86: Stefik, M.J., Bobrow, D.G., and Kahn, K.M. "Integrating Access-Oriented Programming into a Multiparadigm Environment." IEEE Software, 3,1, IEEE Press, January, 1986, 10- 18.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Audio input/output

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology; Interaction styles (e.g., commands, menus, forms, direct manipulation); Input devices and strategies (e.g., mouse, touchscreen)

General Terms:
Design, Human Factors, Languages

Keywords:
GUIs, X, auditory interfaces, multimodal interfaces, visual impairment