Translating keyword commands into executable code

ABSTRACT

Modern applications provide interfaces for scripting, but many users do not know how to write script commands. However, many users are familiar with the idea of entering keywords into a web search engine. Hence, if a user is familiar with the vocabulary of an application domain, we anticipate that they could write a set of keywords expressing a command in that domain. For instance, in the web browsing domain, a user might enter <B>click search button</B>. We call expressions of this form keyword commands, and we present a novel approach for translating keyword commands directly into executable code. Our prototype of this system in the web browsing domain translates <B>click search button</B> into the Chickenfoot code <B>click(findButton("search"))</B>. This code is then executed in the context of a web browser to carry out the effect. We also present an implementation of this system in the domain of Microsoft Word. A user study revealed that subjects could use keyword commands to successfully complete 90% of the web browsing tasks in our study without instructions or training. Conversely, we would expect users to complete close to 0% of the tasks if they had to guess the underlying JavaScript commands with no instructions or training.

REFERENCES

• 1: Apple Computer. Automator. http://www.apple.com/macosx/features/automator/, accessed June 28, 2006.
• 2: Bruce W. Ballard , Alan W. Biermann, Programming in natural language: “NLC” as a prototype, Proceedings of the 1979 annual conference, p.228-237, January 1979 [doi>10.1145/800177.810072]
• 3: Amy Bruckman, Community Support for Constructionist Learning, Computer Supported Cooperative Work, v.7 n.1-2, p.47-86, Jan.1, 1998 [doi>10.1023/A:1008684120893]
• 4: Amy Bruckman , Elizabeth Edwards, Should we leverage natural-language knowledge? An analysis of user errors in a natural-language-style programming language, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.207-214, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303040]
• 5: Margaret Burnett , Curtis Cook , Gregg Rothermel, End-user software engineering, Communications of the ACM, v.47 n.9, September 2004 [doi>10.1145/1015864.1015889]
• 6: Michael Bolin , Matthew Webber , Philip Rha , Tom Wilson , Robert C. Miller, Automation and customization of rendered web pages, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095062]
• 7: Allen Cypher , Daniel C. Halbert , David Kurlander , Henry Lieberman , David Maulsby , Brad A. Myers , Alan Turransky, Watch what I do: programming by demonstration, MIT Press, Cambridge, MA, 1993
• 8: Martin Erwig , Robin Abraham , Irene Cooperstein , Steve Kollmansberger, Automatic generation and maintenance of correct spreadsheets, Proceedings of the 27th international conference on Software engineering, p.136-145, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062494]
• 9: Boris Katz , Sue Felshin , Deniz Yuret , Ali Ibrahim , Jimmy J. Lin , Gregory Marton , Alton Jerome McFarland , Baris Temelkuran, Omnibase: Uniform Access to Heterogeneous Data for Question Answering, Proceedings of the 6th International Conference on Applications of Natural Language to Information Systems-Revised Papers, p.230-234, June 27-28, 2002
• 10: Pausch, R., et al. Alice: A Rapid Prototyping System for 3D Graphics. IEEE Computer Graphics and Applications, 15(3), 8--11, May 1995.
• 11: Caitlin Kelleher , Randy Pausch, Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers, ACM Computing Surveys (CSUR), v.37 n.2, p.83-137, June 2005 [doi>10.1145/1089733.1089734]
• 12: Andrew J. Ko , Brad A. Myers, Designing the whyline: a debugging interface for asking questions about program behavior, Proceedings of the SIGCHI conference on Human factors in computing systems, p.151-158, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985712]
• 13: Andrew J. Ko , Brad A. Myers , Htet Htet Aung, Six Learning Barriers in End-User Programming Systems, Proceedings of the 2004 IEEE Symposium on Visual Languages - Human Centric Computing (VLHCC'04), p.199-206, September 26-29, 2004 [doi>10.1109/VLHCC.2004.47]
• 14: Lieberman, H., Ed. Your Wish is My Command: Programming By Example. Morgan Kaufmann, San Francisco, CA, 2001.
• 15: Hugo Liu , Henry Lieberman, Programmatic semantics for natural language interfaces, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056975]
• 16: Miller, L., Natural Language Programming: Styles, Strategies, and Contrasts. IBM Systems Journal, 1981.
• 17: Miller, P., Pane, J., Meter, G., and Vorthmann, S. Evolution of Novice Programming Environments: The Structure Editors of Carnegie Mellon University. Interactive Learning Environments, 4(2), 140--158, 1994.
• 18: David Mandelin , Lin Xu , Rastislav Bodík , Doug Kimelman, Jungloid mining: helping to navigate the API jungle, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 19: Brad A. Myers , John F. Pane , Andy Ko, Natural programming languages and environments, Communications of the ACM, v.47 n.9, September 2004 [doi>10.1145/1015864.1015888]
• 20: Amit Phalgune , Cory Kissinger , Margaret Burnett , Curtis Cook , Laura Beckwith , Joseph R. Ruthruff, Garbage in, Garbage out? An Empirical Look at Oracle Mistakes by End-User Programmers, Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'05), p.45-52, September 20-24, 2005 [doi>10.1109/VLHCC.2005.40]
• 21: Porter, M., An algorithm for suffix stripping, Program, 14(3), pp 130--137, 1980.
• 22: David Price , Ellen Rilofff , Joseph Zachary , Brandon Harvey, NaturalJava: a natural language interface for programming in Java, Proceedings of the 5th international conference on Intelligent user interfaces, p.207-211, January 09-12, 2000, New Orleans, Louisiana, United States [doi>10.1145/325737.325845]
• 23: Jean E. Sammet, The use of English as a programming language, Communications of the ACM, v.9 n.3, p.228-230, March 1966 [doi>10.1145/365230.365274]
• 24: Tim Teitelbaum , Thomas Reps, The Cornell program synthesizer: a syntax-directed programming environment, Communications of the ACM, v.24 n.9, p.563-573, Sept. 1981 [doi>10.1145/358746.358755]
• 25: Susan Wiedenbeck , Alec Engebretson, Comprehension Strategies of End-User Programmers in an Event-Driven Application, Proceedings of the 2004 IEEE Symposium on Visual Languages - Human Centric Computing (VLHCC'04), p.207-214, September 26-29, 2004 [doi>10.1109/VLHCC.2004.12]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.6 Programming Environments
          Subjects: Interactive environments
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

General Terms:
Algorithms, Design, Experimentation, Human Factors, Languages, Standardization

Keywords:
command languages, end-user programming, natural language processing, web automation