Automated and hand-coded measurement of deliberative quality in online policy discussions

ABSTRACT

A number of research projects are now underway to assess how IT tools and deliberative techniques could enhance public input into government at all levels. The success of such projects depends to an important degree on objective measurement of the quality of discussion under different utilizations of tools and techniques. This paper explores the value of two techniques for determining the quality of policy discussion in the context of federal agency rulemaking: a human-coded technique and a technique involving statistical bootstrapping of natural language processing data. The hand-coded technique utilizes counts of verbal behaviors that are indicative of intellectual engagement. This includes such behaviors as raising questions, disagreeing, and introducing new topics. The automated technique develops a measure of sophistication of reasoning based on significant co-occurrence of concepts in participants' speech. Such co-occurrence clarifies the network of conceptual relations utilized by participants. The more connections participants make between policy-relevant concepts, the more sophisticated their speech should be. Data consist of discussion of the federal rulemaking issue of network neutrality regulation by a sample of 53 volunteers. Correlations and ordinary least squares regression find that the conceptual connection and hand-coded measures, despite being qualitatively quite different, significantly predict each other and several other measures of sophistication, including two indicators of network neutrality knowledge and sophistication of views of government.

REFERENCES

• 1: Yochai Benkler, The Wealth of Networks: How Social Production Transforms Markets and Freedom, Yale University Press, New Haven, CT, 2006
• 2: Button, M. and Ryfe, D. M. What Can We Learn from the Practice of Deliberative Democracy? In J. Gastil and P. Levine, eds., The Deliberative Democracy Handbook: Strategies for Effective Civic Engagement in the Twenty-First Century. Jossey-Bass, San Francisco, 2005, 20--34.
• 3: Cunningham, H., Maynard, D., Bontcheva, K., and Tablan, V. Gate: A Framework and Graphical Development Environment for Robust NLP Tools and Applications. Proceedings of the 40th Annual Meeting of the ACL, (2002).
• 4: Dahlberg, L. The Internet and Democratic Discourse: Exploring The Prospects of Online Deliberative Forums Extending the Public Sphere. Information, Communication and Society 4, 4 (2001), 615--633.
• 5: Dyson, S. B. Text Annotation and the Cognitive Architecture of Political Leaders: British Prime Ministers from 1945--2008. Journal of Information Technology&Politics 5, 1, 7--18.
• 6: Efron, B. and Tibshirani, R. J. An Introduction to the Bootstrap. Chapman&Hall, New York, 1993.
• 7: Fishkin, J. S. Democracy and Deliberation: New Directions for Democratic Reform. Yale University Press, New Haven, 1991.
• 8: Gastil, J. Political Communication and Deliberation. Sage Publications, Inc, 2008.
• 9: Graham, T. and Witschge, T. In Search of Online Deliberation: Towards a New Method for Examining the Quality of Online Discussions. Communications (Sankt Augustin) 28, 2 (2003), 173--204.
• 10: Habermas, J. The Theory of Communicative Action, Volume Two: Lifeworld and System: A Critique of Functionalist Reason. Beacon Press, Boston, 1984.
• 11: Kriplean, T. Designing Mediating Spaces Between Citizens and Government.
• 12: Luskin, R. C., Fishkin, J. S., and Jowell, R. Considered Opinions: Deliberative Polling in Britain. British Journal of Political Science 32, 3 (2002), 455--488.
• 13: Muhlberger, P. Human Agency and the Revitalization of the Public Sphere. Political Communication 22, 2 (2005), 163--178.
• 14: Muhlberger, P. and Weber, L. M. Lessons from the Virtual Agora Project: The Effects of Agency, Identity, Information, and Deliberation on Political Knowledge. Journal of Public Deliberation (available at http://services.bepress.com/jpd/) 2, 1 (2006), 1--39.
• 15: O'Connor, R. E., Bord, R. J., Yarnal, B., and Wiefek, N. Who Wants to Reduce Greenhouse Gas Emissions? Social Science Quarterly 83, 1 (2002), 1--17.
• 16: Petty, R. E., Wegener, D. T., and Fabrigar, L. R. Attitudes and Attitude Change. Annual Review of Psychology 48, (1997), 609--647.
• 17: Howard Rheingold, The Virtual Community: Homesteading on the Electronic Frontier, MIT Press, Cambridge, MA, 2000
• 18: Rottenberg, A. T. The Structure of Argument. Bedford/St. Martin's, 2002.
• 19: Sears, D. O. and Citrin, J. Tax Revolt: Something for Nothing in California. Harvard University Press, Cambridge, Massachusetts, 1982.
• 20: Stromer-Galley, J. Measuring Deliberation's Content: A Coding Scheme. Journal of Public Deliberation 3, 1 (2007).
• 21: Suedfeld, P., Tetlock, P. E., and Streufert, S. Conceptual / Integrative Complexity. In C. P. Smith, J. W. Atkinson and E. al, eds., Motivation and Personality: Handbook of Thematic Content Analysis. Cambridge University Press, New York, NY, US, 1992.
• 22: Tetlock, P. E. Cognitive Structural Analysis of Political Rhetoric: Methodological and Theoretical Issues. In S. Iyengar, ed., Explorations in Political Psychology. Duke University Press, Durham, NC, 1993, 380--405.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY
      K.4.0 General

General Terms:
Experimentation, Human Factors, Measurement, Standardization

Keywords:
deliberation, measurement, natural language processing, political discussion, public engagement, rulemaking, sophistication