Evaluating assistance of natural language policy authoring

ABSTRACT

The goal of the research study reported here was to investigate policy authors' ability to take descriptions of changes to policy situations and author high-quality, complete policy rules that would parse with high accuracy. As a part of this research, we investigated ways in which we could assist policy authors in writing policies. This paper presents the results of a user study on the effectiveness of providing syntax highlighting in a natural language policy authoring interface. While subjects liked the new interface, they showed no improvement in accuracy when writing rules. We discuss our results in terms of a three phase authoring process that users move through when authoring or modifying policies. We describe this process, discuss why and how our interface failed to support it and make recommendations to designers on how to better support this process.

REFERENCES

• 1: Alamargot, D. and Chanquoy, L., Through the Models of Writing. Kluwer Academic Publishers, Dordrecht, The Netherlands, 2001.
• 2: Annie I. Anton , Qingfeng He , David L. Baumer, Inside JetBlue's Privacy Policy Violations, IEEE Security and Privacy, v.2 n.6, p.12-18, November 2004 [doi>10.1109/MSP.2004.103]
• 3: Breaux, T. D., and Anton, A. I. Analyzing Goal Semantics for Rights, Permissions, and Obligations. Technical Report #TR-2005-08, North Carolina State University, Department of Computer Science, February 15, 2005.
• 4: David Baumer , Julia Brande Earp , Fay Cobb Payton, Privacy of medical records: IT implications of HIPAA, ACM SIGCAS Computers and Society, v.30 n.4, p.40-47, December 2000 [doi>10.1145/572260.572261]
• 5: Carolyn A. Brodie , Clare-Marie Karat , John Karat, An empirical study of natural language parsing of privacy policy rules using the SPARCLE policy workbench, Proceedings of the second symposium on Usable privacy and security, July 12-14, 2006, Pittsburgh, Pennsylvania [doi>10.1145/1143120.1143123]
• 6: Kami Brooks, Migrating to role-based access control, Proceedings of the fourth ACM workshop on Role-based access control, p.71-81, October 28-29, 1999, Fairfax, Virginia, United States [doi>10.1145/319171.319178]
• 7: Hayes, J. R. and Flower, L. S. Identifying the organization of writing processes. In L. W. Gregg, and E. R. Steinberg (Eds.), Cognitive processes in writing: An interdisciplinary approach (pp. 3--30). Hillsdale, NJ: Lawrence Erlbaum Associates, 1980.
• 8: Holtzblatt, K. and Beyer, H., Contextual Design. Morgan Kaufmann, San Francisco. 1998.
• 9: Clare-Marie Karat , John Karat , Carolyn Brodie , Jinjuan Feng, Evaluating interfaces for privacy policy rule authoring, Proceedings of the SIGCHI conference on Human Factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1124772.1124787]
• 10: Clare-Marie Karat , John Karat , Carolyn Brodie, Editorial: why HCI research in privacy and security is critical now, International Journal of Human-Computer Studies, v.63 n.1-2, p.1-4, July 2005 [doi>10.1016/j.ijhcs.2005 04.016]
• 11: John Karat , Clare-Marie Karat , Carolyn Brodie , Jinjuan Feng, Privacy in information technology: designing to enable privacy policy management in organizations, International Journal of Human-Computer Studies, v.63 n.1-2, p.153-174, July 2005 [doi>10.1016/j.ijhcs.2005.04.011]
• 12: Günter Karjoth , Matthias Schunter, A Privacy Policy Model for Enterprises, Proceedings of the 15th IEEE workshop on Computer Security Foundations, p.271, June 24-26, 2002
• 13: H. Jeff Smith, Privacy policies and practices: inside the organizational maze, Communications of the ACM, v.36 n.12, p.104-122, Dec. 1993 [doi>10.1145/163298.163349]
• 14: Reeder, R., Karat, C., Karat, J. and Brodie, C., Usability challenges in security and privacy policy-authoring interfaces. INTERACT, 2007.
• 15: L. F. Rubin, Syntax-Directed Pretty Printing A First Step Towards a Syntax-Directed Editor, IEEE Transactions on Software Engineering, v.9 n.2, p.119-127, March 1983 [doi>10.1109/TSE.1983.236456]
• 16: Keith Vertanen , Per Ola Kristensson, On the benefits of confidence visualization in speech recognition, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357288]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

Additional Classification:
  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY
      K.4.1 Public Policy Issues

General Terms:
Design, Human Factors, Security

Keywords:
design process, natural language interfaces, privacy, privacy policies, social and legal issues, syntax highlighting