Should we leverage natural-language knowledge? An analysis of user errors in a natural-language-style programming language

ABSTRACT

Should programming languages use natural-language-like syntax? Under what circumstances? What sorts of errors do novice programmers make? Does using a natural- language-like programming language lead to user errors? In this study, we read the entire online interactions of sixteen children who issued a total of 35,047 commands on MOOSE Crossing, an educational MUD for children, We counted and categorized the errors made. A total d 2,970 errors were observed. We define natural-language errors as those errors in which the user failed to distinguish between English and code, issuing an incorrect command that was more English-like than the correct one. A total of 314 natural-language errors were observed. In most of those errors, the child was able to correct the problem either easily (41.1% of the time) or with some effort (20.7%). Natural-language errors were divided into five categories. In order from most to least frequent, they are: syntax errors, guessing a command name by supplying an arbitrary English word, literal interpretation of metaphor, assuming the system is keeping more state information than is actually the case, and errors of operator precedence and combination. We believe that these error rates are within acceptable limits, and conclude that leveraging users natural-language knowledge is for many applications an effective strategy for designing end-user-programming languages.

REFERENCES

• 1: Hoopper, G.M., Keynote Address, in History of Programming Languages, R.L. Wexelblat, Editor. 1981, Academic Press: New York. p. 7-20.
• 2: Sammet, J., The Early History of COBOL, in History of Programming Languages, R. Wexelblat, Editor. 1981, Academic Press: New York.
• 3: James G. Spohrer , Elliot Soloway, Analyzing the high frequency bugs in novice programs, Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, p.230-251, June 1986, Washington, D.C., United States
• 4: Boulay, B.D., Some Difficulties of Learning to Program, in Studying the Novice Programmer, E. Soloway and J.C. Spohrer, Editors. 1989, Lawrence Erlbaum Associates: Hillsdale, NJ. p. 283-299.
• 5: D. Goodman, The complete Hypercard handbook, Bantam Books, Inc., New York, NY, 1988
• 6: Kaehler, T., 1996, personal communication.
• 7: Bartle, R., Interactive Multi-User Computer Games. 1990, MUSE Ltd: ftp://ftp.lambda.moo.mud.org/pub/MOO/papers/mudxeport.txt
• 8: Bruckman, A. and M. Resnick, The MediaMO0 Project: Constructionism and Professional Community. Convergence, 1995.1(1): p. 94-109.
• 9: Glusman, G., E. Mercer, and I. Rubin, Real-time Collaboration On the Internet: BioMO0, the Biologists' Virtual Meeting Place., in lnternet for the Molecular Biologist., S.R. Swindell, R.R. Miller, andG.S.A. Myers, Editors. 1996, Horizon Scientific Press: Norfolk, UK.
• 10: Van Buren, D., et al., The AstroVR Co,Uaboratory, in Astronomical Data Analysis Software and Systems IV, R. Hanish and H. Payne, Editors. 1994, Astronomical Society of the Pacific: . San Francisco.
• 11: Vicki O‘Day , Daniel Bobrow , Kimberly Bobrow , Mark Shirley , Billie Hughes , Jim Walters, Moving Practice: From Classrooms to MOO Rooms, Computer Supported Cooperative Work, v.7 n.1-2, p.9-45, Jan.1, 1998 [doi>10.1023/A:1008632104055]
• 12: Amy Bruckman, Community Support for Constructionist Learning, Computer Supported Cooperative Work, v.7 n.1-2, p.47-86, Jan.1, 1998 [doi>10.1023/A:1008684120893]
• 13: Amy Susan Bruckman , Mitchel Resnick, Moose crossing: construction, community, and learning in a networked virtual world for kids, 1997
• 14: Curtis, P. Mudding: Social Phenomena in Text- Based Virtual Realities. in DIAC. 1992. Berkeley, CA: ftp://ftp.lambda.moo.mud.org/pub/MOO/papers/DIAC92.txt
• 15: Curtis, P., LambdaMO0 Programmer's Manual. 1993: ftp ://ftp. lambda.moo.mud, org/pub/M OOfProgrammersManual.txt
• 16: Curtis, P. and D. Nichols. MUDs Grow Up: Social Virtual Reality in the Real World'. in Third International Conference on Cyberspace. 1993. Austin, Texas: ftp://ftp.lambda.moo.mud.org/pub/MOO/papers/MUDsGrowUp.txt
• 17: Seymour Papert, Mindstorms: Children, computers, and powerful ideas, Basic Books, Inc., New York, NY, 1980
• 18: Miller, L.A., Natural language programming: Styles, strategies, and constrasts: IBM Systems Journal, 1981. 211(2): p. 184-215.
• 19: Bonar, J. and E. Soloway, Preprogramming Knowledge: A Major Source of Misconceptions in Novice Programmers, in Studying the Novice Programmer, E. Soloway and J. Spohrer, Editors. 1989, Lawrence Erlbaum Associates: Hillsdale, NJ.
• 20: Pea, R.D., Language-Independent Conceptual "Bugs" in Novice Programming. Journal af Educational Computing Research, 1986. 2(1): p. 25- 36.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE

General Terms:
Design, Languages, Theory

Keywords:
end-user programming, natural language, novice programming, programming language design