Exploring educational standard alignment: in search of 'relevance'

ABSTRACT

The growing availability of online K-12 curriculum is increasing the need for meaningful alignment of this curriculum with state-specific standards. Promising automated and semi-automated alignment tools have recently become available. Unfortunately, recent alignment evaluation studies report low inter-rater reliability, e.g., 32% with two raters and 35 documents. While these results are in line with studies in other domains, low reliability makes it difficult to accurately train automatic systems and complicates comparison of different services. We propose that inter-rater reliability of broadly defined, abstract concepts such as 'alignment' or 'relevance' must be expected to be low due to the real-world complexity of teaching and the multidimensional nature of the curricular documents. Hence, we suggest decomposing these concepts into less abstract, more precise measures anchored in the daily practice of teaching.

This article reports on the integration of automatic alignment results into the interface of the Teach Engineering collection and on an evaluation methodology intended to produce more consistent document relevance ratings. Our results (based on 14 raters x 6 documents) show high inter-rater reliability (61 - 95%) on less abstract relevance dimensions while scores on the overall 'relevance' concept are (as expected) lower (64%). Despite a relatively small sample size, regression analysis of our data resulted in an explanatory (R² = .75) and statistically stable (p-values < .05) model for overall relevance as indicated by matching concepts, related background material, adaptability to grade level, and anticipated usefulness of exercises. Our results suggest that more detailed relevance evaluation which includes several dimensions of relevance would produce better data for comparing and training alignment tools.

REFERENCES

• 1: Judit Bar-Ilan , Kevin Keenoy , Eti Yaari , Mark Levene, User rankings of search engine results, Journal of the American Society for Information Science and Technology, v.58 n.9, p.1254-1266, July 2007 [doi>10.1002/asi.20608]
• 2: Devaul, H., Diekema, A. R., Ostwald, J. (2007) Computer-assisted Assignment of Educational Standards Using Natural Language Processing. Paper presented at the Annual Meeting of the National Science Digital Library (NSDL). Arlington, VA.
• 3: Anne R. Diekema , Jiangping Chen, Experimenting with the automatic assignment of educational standards to digital library content, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065436]
• 4: Gateway (2007). NSDL:ASN Achievement Standards Network. Available: http://www.thegateway.org/asn.
• 5: Greenwood, J. D. (1982) On the Relation Between Laboratory Experiments and Social Behavior: Causal Explanation and Generalization. Journal for the Theory of Social Behavior. 12. 225--250.
• 6: Carl Lagoze , Sandy Payette , Edwin Shin , Chris Wilper, Fedora: an architecture for complex objects and their relationships, International Journal on Digital Libraries, v.6 n.2, p.124-138, April 2006 [doi>10.1007/s00799-005-0130-3]
• 7: Lagoze, C., Kraft, D. B., Payette, S., Jesuroga, S. (2005) What Is a Digital Library Anymore, Anyway? Beyond Search and Access in the NSDL. D-Lib Magazine. 11.
• 8: Lazarsfeld, P. F. (1954) A Conceptual Introduction to Latent Structure Analysis. In: Lazarsfeld, P. F. (Ed.) Mathematical Thinking in the Social Sciences. Free Press. New York. NY.
• 9: Levins, R., Lewontin, R. C. (1980) Dialectics and Reductionalism in Ecology. Synthese. 43. 47--78.
• 10: Byron B. Marshall , René F. Reitsma , Martha N. Cyr, Standards or semantics for curriculum search?, Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, June 18-23, 2007, Vancouver, BC, Canada [doi>10.1145/1255175.1255210]
• 11: Noble, W. G. (1981) Gibsonian Theory and the Pragmatist Perspective. Journal for the Theory of Social Behavior. 11. 65--85.
• 12: Saracevic, T. (1975) Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Journal of the American Society of Information Science. 26(6). 321--343
• 13: Tefko Saracevic, Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance, Journal of the American Society for Information Science and Technology, v.58 n.13, p.1915-1933, November 2007 [doi>10.1002/asi.v58:13]
• 14: Tefko Saracevic, Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance, Journal of the American Society for Information Science and Technology, v.58 n.13, p.2126-2144, November 2007 [doi>10.1002/asi.v58:13]
• 15: Van der Smagt (1985) Definieren en Relateren in Sociaal Wetenschappelijk Onderzoek. (Definitions and Relations in Social Science Research). Dissertation. University of Nijmegen, The Netherlands.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Computer-assisted instruction (CAI)

General Terms:
Design, Experimentation, Human Factors, Measurement, Reliability, Theory

Keywords:
context-specific measurement, curriculum-standard alignment, digital library, inter-rater reliability, reification, relevance, social science theory