Comparing evaluation techniques for text readability software for adults with intellectual disabilities

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback
Take a look at the new version of this page: [ beta version ]. Tell us what you think.

Comparing evaluation techniques for text readability software for adults with intellectual disabilities

Full text

Pdf (827 KB)

Source

ACM SIGACCESS Conference on Computers and Accessibility archive
Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility table of contents

Pittsburgh, Pennsylvania, USA

SESSION: Cognitive accessibility table of contents

Pages: 3-10

Year of Publication: 2009

ISBN:978-1-60558-558-1

Authors

Matt Huenerfauth	The City University of New York, Queens College, Flushing, NY, USA
Lijun Feng	The City University of New York, Graduate Center, New York, NY, USA
Noémie Elhadad	Columbia University, New York, NY, USA

Sponsor

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 22, Downloads (12 Months): 51, Citation Count: 0

Additional Information:

abstract references index terms collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1639642.1639646 What is a DOI?

ABSTRACT

In this paper, we compare alternative techniques for evaluating a software system for simplifying the readability of texts for adults with mild intellectual disabilities (ID). We introduce our research on the development of software to automatically simplify news articles, display them, and read them aloud for adults with ID. Using a Wizard-of-Oz prototype, we conducted experiments with a group of adults with ID to test alternative formats of questions to measure comprehension of the information in the news articles. We have found that some forms of questions work well at measuring the difficulty level of a text: multiple-choice questions with three answer choices, each illustrated with clip-art or a photo. Some types of questions do a poor job: yes/no questions and Likert-scale questions in which participants report their perception of the text's difficulty level. Our findings inform the design of future evaluation studies of computational linguistic software for adults with ID; this study may also be of interest to researchers conducting usability studies or other surveys with adults with ID.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Carroll, J., Minnen, G., Pearce, D., Canning, Y., Devlin, S., Tait, J. 1999. Simplifying text for language-impaired readers. In Proceedings of EACL'99 Poster, p. 269.
	2	Chall, J.S., and Dale, E. 1995. Readability revisited: the new Dale-Chall readability formula. Cambridge: Brookline.
	3	Cohen, S.A., Steinberg, J.E. 1983. Effects of Three Types of Vocabulary on Readability of Intermediate Grade Science Textbooks: An Application of Finn's Transfer Feature Theory. Read Res Q, 19(1):86--101.
	4	Davies, D., Stock, S. and Wehmeyer, M. 2001. Enhancing Independent Internet Access for Individuals with Mental Retardation through Use of a Specialized Web Browser: A Pilot Study. Educ Train Ment Retard Dev Disabil, 36(1):107--113.
	5	Davies D., Stock S., King L., and Wehmeyer M. 2008. "Moby-Dick Is My Favorite:" Evaluating a Cognitively Accessible Portable Reading System for Audiobooks for Individuals with Intellectual Disability. Intellectual and Developmental Disabilities. 46(4):290--298.
	6	Davison, A., and Kantor, R. 1982. On the failure of readability formulas to define readable texts: A case study from adaptations. Read Res Q, 17(2):187--209.
	7	Melissa Dawe, Understanding mobile phone requirements for young adults with cognitive disabilities, Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, October 15-17, 2007, Tempe, Arizona, USA [doi>10.1145/1296843.1296874]
	8	Drew, C.J., and Hardman, M.L. 2004. Mental retardation: A lifespan approach to people with intellectual disabilities (8th ed.). Columbus, OH: Merrill.
	9	J. Feng , J. Lazar , L. Kumin , A. Ozok, Computer usage by young individuals with down syndrome: an exploratory study, Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility, October 13-15, 2008, Halifax, Nova Scotia, Canada [doi>10.1145/1414471.1414480]
	10	Lijun Feng , Noémie Elhadad , Matt Huenerfauth, Cognitively motivated features for readability assessment, Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, p.229-237, March 30-April 03, 2009, Athens, Greece
	11	Stephen Fickas , Craig Pataky , Zebin Chen, DuckCall:: tackling the first hundred yards problem, Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, October 23-25, 2006, Portland, Oregon, USA [doi>10.1145/1168987.1169056]
	12	Fowler, A.E. 1998. Language in mental retardation. In Burack, Hodapp, and Zigler (Eds.), Handbook of Mental Retardation and Development. Cambridge, UK: Cambridge Univ. Press, pp. 290--333.
	13	Gray, W. S. and B. Leary. 1935. What makes a book readable. Chicago: Chicago University Press.
	14	Heilman, M., Collins-Thompson, K., Callan, J., and Eskenazi, M. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Proceedings of NAACL'07, pp. 460--467.
	15	Hickson-Bilsky, L. 1985. Comprehension and mental retardation. International Review of Research in Mental Retardation, 13:215--246, St. Louis: Elsevier.
	16	Matt Huenerfauth, Evaluation of a psycholinguistically motivated timing model for animations of american sign language, Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility, October 13-15, 2008, Halifax, Nova Scotia, Canada [doi>10.1145/1414471.1414496]
	17	Jones, F.W., Long, K., Finlay, W.M. 2006. Assessing the reading comprehension of adults with learning disabilities. J Intel Disabil Res, 50(6):410--418, Malden, MA: Blackwell.
	18	Jones, F.W., Long K., and Finlay W.M.L. 2007. Symbols can improve the reading comprehension of adults with learning disabilities. J Intel Disabil Res, 51(7):545--550.
	19	Katims, D.S. 2000. Literacy instruction for people with mental retardation: Historical highlights and contemporary analysis. Educ Train Ment Retard Dev Disabil, 35(1):3--15.
	20	Kincaid, J., Fishburne, R., Rodgers, R., and Chisson, B. 1975. Derivation of new readability formulas for navy enlisted personnel. Technical report, Research Branch Report 8-75, U.S. Naval Air Station.
	21	Kintsch, W. 1998. Comprehension: A paradigm for cognition. New York: Cambridge University Press.
	22	Auli Lepistö , Saila Ovaska, Usability evaluation involving participants with cognitive disabilities, Proceedings of the third Nordic conference on Human-computer interaction, p.305-308, October 23-27, 2004, Tampere, Finland [doi>10.1145/1028014.1028061]
	23	Edmund LoPresti , Ned Kirsch , Richard Simpson , Debra Schreckenghost, Solo: interactive task guidance, Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility, October 09-12, 2005, Baltimore, MD, USA [doi>10.1145/1090785.1090824]
	24	Maki, R.H., and Berry, S.L. 1984. Metacomprehension of text material. Journal of Experimental Psychology: Learning, Memory, and Cognition. 10(4):663--679.
	25	Perfetti, C., Lesgold, A. 1977. Discourse comprehension and sources of individual differences. In M. A. Just,&P. A. Carpenter (eds.), Cognitive processes in comprehension. Hillsdale, NJ: Lawrence Erlbaum Assoc.
	26	Sarah E. Petersen , Mari Ostendorf, A machine learning approach to reading level assessment, Computer Speech and Language, v.23 n.1, p.89-106, January, 2009 [doi>10.1016/j.csl.2008.04.003]
	27	Luo Si , Jamie Callan, A statistical model for scientific readability, Proceedings of the tenth international conference on Information and knowledge management, October 05-10, 2001, Atlanta, Georgia, USA [doi>10.1145/502585.502695]
	28	Jeon Small , Pamela Schallau , Karen Brown , Richard Appleyard, Web accessibility for people with cognitive disabilities, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057024]
	29	SpheNet. 2007. TTSReader, (accessed December 8, 2007), http://www.sphenet.com/TTSReader/
	30	TheArcLink. 2003. The Desk Launches in 11 States for People with Disabilities, (accessed October 29, 2008), http://www.thearclink.org/news/article.asp?ID=591
	31	U.S. Census Bureau. 2000. Projections of the total resident population by five-year age groups and sex, with special age categories: Middle series 2025-2045. Washington: Census Bureau, Populations Projections Prog., Population Division.
	32	Williams, S., and Reiter, E. 2005. Generating readable texts for readers with low basic skills. In Proc ENLG'05, pp. 140--147.
	33	Wong, B., Jones, W. 1982. Increasing Metacomprehension in Learning Disabled and Normally Achieving Students through Self-Questioning Training. Learn Disabil Q, 5(3):228--240.