Comparing evaluation techniques for text readability software for adults with intellectual disabilities

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

Comparing evaluation techniques for text readability software for adults with intellectual disabilities

Full text

Pdf (827 KB)

Source

ACM SIGACCESS Conference on Computers and Accessibility archive
Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility table of contents

Pittsburgh, Pennsylvania, USA

SESSION: Cognitive accessibility table of contents

Pages 3-10

Year of Publication: 2009

ISBN:978-1-60558-558-1

Authors

Matt Huenerfauth	The City University of New York, Queens College, Flushing, NY, USA
Lijun Feng	The City University of New York, Graduate Center, New York, NY, USA
Noémie Elhadad	Columbia University, New York, NY, USA

Sponsor

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 23, Downloads (12 Months): 23, Citation Count: 0

Additional Information:

abstract references index terms

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/1639642.1639646 What is a DOI?

ABSTRACT

In this paper, we compare alternative techniques for evaluating a software system for simplifying the readability of texts for adults with mild intellectual disabilities (ID). We introduce our research on the development of software to automatically simplify news articles, display them, and read them aloud for adults with ID. Using a Wizard-of-Oz prototype, we conducted experiments with a group of adults with ID to test alternative formats of questions to measure comprehension of the information in the news articles. We have found that some forms of questions work well at measuring the difficulty level of a text: multiple-choice questions with three answer choices, each illustrated with clip-art or a photo. Some types of questions do a poor job: yes/no questions and Likert-scale questions in which participants report their perception of the text's difficulty level. Our findings inform the design of future evaluation studies of computational linguistic software for adults with ID; this study may also be of interest to researchers conducting usability studies or other surveys with adults with ID.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Carroll, J., Minnen, G., Pearce, D., Canning, Y., Devlin, S., Tait, J. 1999. Simplifying text for language-impaired readers. In Proceedings of EACL'99 Poster, p. 269.
	2	Chall, J.S., and Dale, E. 1995. Readability revisited: the new Dale-Chall readability formula. Cambridge: Brookline.
	3	Cohen, S.A., Steinberg, J.E. 1983. Effects of Three Types of Vocabulary on Readability of Intermediate Grade Science Textbooks: An Application of Finn's Transfer Feature Theory. Read Res Q, 19(1):86--101.
	4	Davies, D., Stock, S. and Wehmeyer, M. 2001. Enhancing Independent Internet Access for Individuals with Mental Retardation through Use of a Specialized Web Browser: A Pilot Study. Educ Train Ment Retard Dev Disabil, 36(1):107--113.
	5	Davies D., Stock S., King L., and Wehmeyer M. 2008. "Moby-Dick Is My Favorite:" Evaluating a Cognitively Accessible Portable Reading System for Audiobooks for Individuals with Intellectual Disability. Intellectual and Developmental Disabilities. 46(4):290--298.
	6	Davison, A., and Kantor, R. 1982. On the failure of readability formulas to define readable texts: A case study from adaptations. Read Res Q, 17(2):187--209.
	7	Dawe, M. 2007. Understanding mobile phone requirements for young adults with cognitive disabilities. In Proceedings of ASSETS'07, New York: ACM Press, pp. 179--186.
	8	Drew, C.J., and Hardman, M.L. 2004. Mental retardation: A lifespan approach to people with intellectual disabilities (8th ed.). Columbus, OH: Merrill.
	9	Feng, J., Lazar, J., Kumin, L., and Ozok, A. 2008. Computer usage by young individuals with down syndrome: an exploratory study. In Proceedings of ASSETS'08, New York: ACM Press, pp. 35--42.
	10	Feng, L., Elhadad, N., Huenerfauth, M. 2009. Cognitively Motivated Features for Readability Assessment. In Proceedings of EACL'09, pp. 229--237.
	11	Fickas, S., Pataky, C., and Chen, Z. 2006. DuckCall: tackling the first hundred yards problem. In Proceedings of ASSETS'06. New York: ACM Press, pp. 283--284.
	12	Fowler, A.E. 1998. Language in mental retardation. In Burack, Hodapp, and Zigler (Eds.), Handbook of Mental Retardation and Development. Cambridge, UK: Cambridge Univ. Press, pp. 290--333.
	13	Gray, W. S. and B. Leary. 1935. What makes a book readable. Chicago: Chicago University Press.
	14	Heilman, M., Collins-Thompson, K., Callan, J., and Eskenazi, M. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Proceedings of NAACL'07, pp. 460--467.
	15	Hickson-Bilsky, L. 1985. Comprehension and mental retardation. International Review of Research in Mental Retardation, 13:215--246, St. Louis: Elsevier.
	16	Huenerfauth, M. 2008. Evaluation of a Psycholinguistically Motivated Timing Model for Animations of American Sign Language. In Proceedings of ASSETS'08, New York: ACM Press, pp. 129--136.
	17	Jones, F.W., Long, K., Finlay, W.M. 2006. Assessing the reading comprehension of adults with learning disabilities. J Intel Disabil Res, 50(6):410--418, Malden, MA: Blackwell.
	18	Jones, F.W., Long K., and Finlay W.M.L. 2007. Symbols can improve the reading comprehension of adults with learning disabilities. J Intel Disabil Res, 51(7):545--550.
	19	Katims, D.S. 2000. Literacy instruction for people with mental retardation: Historical highlights and contemporary analysis. Educ Train Ment Retard Dev Disabil, 35(1):3--15.
	20	Kincaid, J., Fishburne, R., Rodgers, R., and Chisson, B. 1975. Derivation of new readability formulas for navy enlisted personnel. Technical report, Research Branch Report 8-75, U.S. Naval Air Station.
	21	Kintsch, W. 1998. Comprehension: A paradigm for cognition. New York: Cambridge University Press.
	22	Lepistö, A., and Ovaska, S. 2004. Usability evaluation involving participants with cognitive disabilities. In Proc of NordiCHI-2004, New York: ACM Press, pp. 305--308.
	23	LoPresti, E., Kirsch, N., Simpson, R., and Schreckenghost, D. 2005. Solo: interactive task guidance. In Proceedings of ASSETS'05, New York: ACM Press, pp. 190--191.
	24	Maki, R.H., and Berry, S.L. 1984. Metacomprehension of text material. Journal of Experimental Psychology: Learning, Memory, and Cognition. 10(4):663--679.
	25	Perfetti, C., Lesgold, A. 1977. Discourse comprehension and sources of individual differences. In M. A. Just, & P. A. Carpenter (eds.), Cognitive processes in comprehension. Hillsdale, NJ: Lawrence Erlbaum Assoc.
	26	Petersen, S.E., Ostendorf, M. 2009. A machine learning approach to reading level assessment. Computer Speech and Language, 23: 89--106, St. Louis: Elsevier.
	27	Si, L., and Callan, J. 2001. A statistical model for scientific readability. In Proc CIKM'01, ACM Press, pp. 574--576.
	28	Small, J., Schallau, P., Brown, K., and Appleyard, R. 2005. Web Accessibility for People with Cognitive Disabilities. In CHI '05 Extended Abstracts, New York: ACM, 1793--1796.
	29	SpheNet. 2007. TTSReader, (accessed December 8, 2007), http://www.sphenet.com/TTSReader/
	30	TheArcLink. 2003. The Desk Launches in 11 States for People with Disabilities, (accessed October 29, 2008), http://www.thearclink.org/news/article.asp?ID=591
	31	U.S. Census Bureau. 2000. Projections of the total resident population by five-year age groups and sex, with special age categories: Middle series 2025-2045. Washington: Census Bureau, Populations Projections Prog., Population Division.
	32	Williams, S., and Reiter, E. 2005. Generating readable texts for readers with low basic skills. In Proc ENLG'05, pp. 140--147.
	33	Wong, B., Jones, W. 1982. Increasing Metacomprehension in Learning Disabled and Normally Achieving Students through Self-Questioning Training. Learn Disabil Q, 5(3):228--240.