Usability of voting systems: baseline data for paper, punch cards, and lever machines

ABSTRACT

In the United States, computer-based voting machines are rapidly replacing other older technologies. While there is potential for this to be a usability improvement, particularly in terms of accessibility, the only way it is possible to know if usability has improved is to have baseline data on the usability of traditional technologies. We report an experiment assessing the usability of punch cards, lever machines, and two forms of paper ballot. There were no differences in ballot completion time between the four methods, but there were substantial effects on error rate, with the paper ballots superior to the other methods as well as an interaction with age of voters. Subjective usability was assessed with the System Usability Scale and showed a slight advantage for bubble-style paper ballots. Overall, paper ballots were found to be particularly usable, which raises important technological and policy issues.

REFERENCES

• 1: Ansolabehere, S., & Stewart, C., III. (2005). Residual votes attributable to technology. Journal of Politics, 67(2), 365--389.
• 2: Benjamin B. Bederson , Bongshin Lee , Robert M. Sherman , Paul S. Herrnson , Richard G. Niemi, Electronic voting system usability issues, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642638]
• 3: Brooke, J. (1996). SUS: A "quick and dirty" usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester & A. L. McClelland (Eds.), Usability Evaluation in Industry (pp. 189--194). London: Taylor and Francis.
• 4: John P. Chin , Virginia A. Diehl , Kent L. Norman, Development of an instrument measuring user satisfaction of the human-computer interface, Proceedings of the SIGCHI conference on Human factors in computing systems, p.213-218, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57203]
• 5: Conrad, F. G., Lewis, B., Peytcheva, E., Traugott, M., Hanmer, M. J., Herrnson, P. S., et al. (2006). The usability of electronic voting systems: Results from a laboratory study. Paper presented at the Midwest Political Science Association, Chicago, IL. April 2006.
• 6: Everett, S. P., Byrne, M. D., & Greene, K. K. (2006). Measuring the usability of paper ballots: Efficiency, effectiveness, and satisfaction. Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society.
• 7: Erik Frøkjær , Morten Hertzum , Kasper Hornbæk, Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.345-352, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/332040.332455]
• 8: Kristen K. Greene , Michael D. Byrne , Sarah P. Everett, A comparison of usability between voting methods, Proceedings of the USENIX/Accurate Electronic Voting Technology Workshop 2006 on Electronic Voting Technology Workshop, p.2-2, August 01, 2006, Vancouver, B.C., Canada
• 9: Herrnson, P. S., Niemi, R. G., Hanmer, M. J., Bederson, B. B., Conrad, F. G., & Traugott, M. (2006). Voters'abilities to cast their votes as intended. Paper presented at the Workshop on the Usability and Security of Electronic Voting System.
• 10: Industry Usability Reporting Project. (2001). Common industry format for usability test reports (ANSI/INCITS 354-2001).
• 11: International Committee for Information Technology Standards. ISO 9241-11. (1998). Ergonomic requirements for office work with visual display terminals (VDT)s part 11. Guidance on usability.
• 12: Kimball, D. C., & Kropf, M. (2005). Ballot design and unrecorded votes on paper-based ballots. Public Opinion Quarterly, 69(4), 508--529.
• 13: Kohno, T., Stubblefield, A., Rubin, A. D., & Wallach, D. S. (2004). Analysis of an electronic voting system. Paper presented at the 2004 IEEE Symposium on Security and Privacy, Oakland, CA. May 2004.
• 14: Laskowski, S. J., Autry, M., Cugini, J., Killam, W., & Yen, J. (2004). Improving the usability and accessibility of voting systems and products. NIST Special Publication 500--256.
• 15: Mebane, W. R. (2004). The wrong man is president! Overvotes in the 2000 presidential election in Florida. Perspectives on Politics, 2(3), 525--535.
• 16: Roth, S. K. (1998). Disenfranchised by design: Voting systems and the election process. Information Design Journal, 9(1), 1--8.
• 17: Abigail J. Sellen , Richard H.R. Harper, The Myth of the Paperless Office, MIT Press, Cambridge, MA, 2003
• 18: Taebel, D. A. (1975). The effect of ballot position on electoral success. American Journal of Political Science, 19(3), 519--526.
• 19: Wand, J. N., Shotts, K. W., Sekhon, J. S., Mebane, W. R., Herron, M. C., & Brady, H. E. (2001). The butterfly did it: The aberrant vote for Buchanan in Palm Beach County, Florida. American Political Science Review, 95(4), 793--810.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Keywords:
DRE, effectiveness, efficiency, subjective usability, voting