research-article

Comparing evaluation techniques for text readability software for adults with intellectual disabilities

Authors:

Matt Huenerfauth,

Noémie ElhadadAuthors Info & Claims

Assets '09: Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility

Pages 3 - 10

https://doi.org/10.1145/1639642.1639646

Published: 25 October 2009 Publication History

Abstract

In this paper, we compare alternative techniques for evaluating a software system for simplifying the readability of texts for adults with mild intellectual disabilities (ID). We introduce our research on the development of software to automatically simplify news articles, display them, and read them aloud for adults with ID. Using a Wizard-of-Oz prototype, we conducted experiments with a group of adults with ID to test alternative formats of questions to measure comprehension of the information in the news articles. We have found that some forms of questions work well at measuring the difficulty level of a text: multiple-choice questions with three answer choices, each illustrated with clip-art or a photo. Some types of questions do a poor job: yes/no questions and Likert-scale questions in which participants report their perception of the text's difficulty level. Our findings inform the design of future evaluation studies of computational linguistic software for adults with ID; this study may also be of interest to researchers conducting usability studies or other surveys with adults with ID.

References

[1]

Carroll, J., Minnen, G., Pearce, D., Canning, Y., Devlin, S., Tait, J. 1999. Simplifying text for language-impaired readers. In Proceedings of EACL'99 Poster, p. 269.

[2]

Chall, J.S., and Dale, E. 1995. Readability revisited: the new Dale-Chall readability formula. Cambridge: Brookline.

[3]

Cohen, S.A., Steinberg, J.E. 1983. Effects of Three Types of Vocabulary on Readability of Intermediate Grade Science Textbooks: An Application of Finn's Transfer Feature Theory. Read Res Q, 19(1):86--101.

[4]

Davies, D., Stock, S. and Wehmeyer, M. 2001. Enhancing Independent Internet Access for Individuals with Mental Retardation through Use of a Specialized Web Browser: A Pilot Study. Educ Train Ment Retard Dev Disabil, 36(1):107--113.

[5]

Davies D., Stock S., King L., and Wehmeyer M. 2008. "Moby-Dick Is My Favorite:" Evaluating a Cognitively Accessible Portable Reading System for Audiobooks for Individuals with Intellectual Disability. Intellectual and Developmental Disabilities. 46(4):290--298.

[6]

Davison, A., and Kantor, R. 1982. On the failure of readability formulas to define readable texts: A case study from adaptations. Read Res Q, 17(2):187--209.

[7]

Dawe, M. 2007. Understanding mobile phone requirements for young adults with cognitive disabilities. In Proceedings of ASSETS'07, New York: ACM Press, pp. 179--186.

Digital Library

[8]

Drew, C.J., and Hardman, M.L. 2004. Mental retardation: A lifespan approach to people with intellectual disabilities (8th ed.). Columbus, OH: Merrill.

[9]

Feng, J., Lazar, J., Kumin, L., and Ozok, A. 2008. Computer usage by young individuals with down syndrome: an exploratory study. In Proceedings of ASSETS'08, New York: ACM Press, pp. 35--42.

Digital Library

[10]

Feng, L., Elhadad, N., Huenerfauth, M. 2009. Cognitively Motivated Features for Readability Assessment. In Proceedings of EACL'09, pp. 229--237.

Digital Library

[11]

Fickas, S., Pataky, C., and Chen, Z. 2006. DuckCall: tackling the first hundred yards problem. In Proceedings of ASSETS'06. New York: ACM Press, pp. 283--284.

Digital Library

[12]

Fowler, A.E. 1998. Language in mental retardation. In Burack, Hodapp, and Zigler (Eds.), Handbook of Mental Retardation and Development. Cambridge, UK: Cambridge Univ. Press, pp. 290--333.

[13]

Gray, W. S. and B. Leary. 1935. What makes a book readable. Chicago: Chicago University Press.

[14]

Heilman, M., Collins-Thompson, K., Callan, J., and Eskenazi, M. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Proceedings of NAACL'07, pp. 460--467.

[15]

Hickson-Bilsky, L. 1985. Comprehension and mental retardation. International Review of Research in Mental Retardation, 13:215--246, St. Louis: Elsevier.

[16]

Huenerfauth, M. 2008. Evaluation of a Psycholinguistically Motivated Timing Model for Animations of American Sign Language. In Proceedings of ASSETS'08, New York: ACM Press, pp. 129--136.

Digital Library

[17]

Jones, F.W., Long, K., Finlay, W.M. 2006. Assessing the reading comprehension of adults with learning disabilities. J Intel Disabil Res, 50(6):410--418, Malden, MA: Blackwell.

[18]

Jones, F.W., Long K., and Finlay W.M.L. 2007. Symbols can improve the reading comprehension of adults with learning disabilities. J Intel Disabil Res, 51(7):545--550.

[19]

Katims, D.S. 2000. Literacy instruction for people with mental retardation: Historical highlights and contemporary analysis. Educ Train Ment Retard Dev Disabil, 35(1):3--15.

[20]

Kincaid, J., Fishburne, R., Rodgers, R., and Chisson, B. 1975. Derivation of new readability formulas for navy enlisted personnel. Technical report, Research Branch Report 8-75, U.S. Naval Air Station.

[21]

Kintsch, W. 1998. Comprehension: A paradigm for cognition. New York: Cambridge University Press.

[22]

Lepistö, A., and Ovaska, S. 2004. Usability evaluation involving participants with cognitive disabilities. In Proc of NordiCHI-2004, New York: ACM Press, pp. 305--308.

Digital Library

[23]

LoPresti, E., Kirsch, N., Simpson, R., and Schreckenghost, D. 2005. Solo: interactive task guidance. In Proceedings of ASSETS'05, New York: ACM Press, pp. 190--191.

Digital Library

[24]

Maki, R.H., and Berry, S.L. 1984. Metacomprehension of text material. Journal of Experimental Psychology: Learning, Memory, and Cognition. 10(4):663--679.

[25]

Perfetti, C., Lesgold, A. 1977. Discourse comprehension and sources of individual differences. In M. A. Just,&P. A. Carpenter (eds.), Cognitive processes in comprehension. Hillsdale, NJ: Lawrence Erlbaum Assoc.

[26]

Petersen, S.E., Ostendorf, M. 2009. A machine learning approach to reading level assessment. Computer Speech and Language, 23: 89--106, St. Louis: Elsevier.

Digital Library

[27]

Si, L., and Callan, J. 2001. A statistical model for scientific readability. In Proc CIKM'01, ACM Press, pp. 574--576.

Digital Library

[28]

Small, J., Schallau, P., Brown, K., and Appleyard, R. 2005. Web Accessibility for People with Cognitive Disabilities. In CHI '05 Extended Abstracts, New York: ACM, 1793--1796.

Digital Library

[29]

SpheNet. 2007. TTSReader, (accessed December 8, 2007), http://www.sphenet.com/TTSReader/

[30]

TheArcLink. 2003. The Desk Launches in 11 States for People with Disabilities, (accessed October 29, 2008), http://www.thearclink.org/news/article.asp?ID=591

[31]

U.S. Census Bureau. 2000. Projections of the total resident population by five-year age groups and sex, with special age categories: Middle series 2025-2045. Washington: Census Bureau, Populations Projections Prog., Population Division.

[32]

Williams, S., and Reiter, E. 2005. Generating readable texts for readers with low basic skills. In Proc ENLG'05, pp. 140--147.

[33]

Wong, B., Jones, W. 1982. Increasing Metacomprehension in Learning Disabled and Normally Achieving Students through Self-Questioning Training. Learn Disabil Q, 5(3):228--240.

Cited By

Säuberli AHolzknecht FHaller PDeilen SSchiffl LHansen-Schirra SEbling S(2024)Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual DisabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642570(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642570
Säuberli AHansen-Schirra SHolzknecht FGutermuth SDeilen SSchiffl LEbling S(2023)Enabling text comprehensibility assessment for people with intellectual disabilities using a mobile applicationFrontiers in Communication10.3389/fcomm.2023.11756258Online publication date: 3-Aug-2023
https://doi.org/10.3389/fcomm.2023.1175625
Colley MKränzle TRukzio E(2022)Accessibility-Related Publication Distribution in HCI Based on a Meta-AnalysisExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519701(1-28)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519701
Show More Cited By

Index Terms

Comparing evaluation techniques for text readability software for adults with intellectual disabilities
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Machine translation
      2. Natural language generation
2. Social and professional topics
  1. Professional topics
    1. Computing profession
      1. Assistive technologies
  2. User characteristics
    1. People with disabilities

Recommendations

Reading difficulty in adults with intellectual disabilities: analysis with a hierarchical latent trait model
ASSETS '10: Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility

In prior work, adults with intellectual disabilities answered comprehension questions after reading texts. We apply a latent trait model to this data to infer the intrinsic difficulty of texts for the participant group. We then analyze the correlation ...
Acceptability of Assistive Technology Promoting Independent Travel of People with Intellectual Disabilities (SAMDI PROJECT): A Focus Group Study from Support Staff
HCI in Mobility, Transport, and Automotive Systems
Abstract
Independent travel is essential for social participation and inclusion in the community. However, several people with intellectual and developmental disabilities restrict their community mobility to the vicinity of their residential area and face ...
ACCESS+: Designing a Museum Application for People with Intellectual Disabilities
Computers Helping People with Special Needs
Abstract
Inclusive solutions are essential to improve the user experience and overall accessibility. They contribute to the independence and participation of people with disabilities and can be designed for a wide variety of contexts. In this paper, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Assets '09: Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility

October 2009

290 pages

ISBN:9781605585581

DOI:10.1145/1639642

General Chair:
Shari Trewin
IBM T. J. Watson Research Center, USA
,
Program Chair:
Kathleen F. McCoy
University of Delaware, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGACCESS: ACM Special Interest Group on Accessible Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASSETS09

Sponsor:

SIGACCESS

ASSETS09: The 11th International ACM SIGACCESS Conference on Computers and Accessibility

October 25 - 28, 2009

Pennsylvania, Pittsburgh, USA

Acceptance Rates

Overall Acceptance Rate 436 of 1,556 submissions, 28%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
718
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)5

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Säuberli AHolzknecht FHaller PDeilen SSchiffl LHansen-Schirra SEbling S(2024)Digital Comprehensibility Assessment of Simplified Texts among Persons with Intellectual DisabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642570(1-11)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642570
Säuberli AHansen-Schirra SHolzknecht FGutermuth SDeilen SSchiffl LEbling S(2023)Enabling text comprehensibility assessment for people with intellectual disabilities using a mobile applicationFrontiers in Communication10.3389/fcomm.2023.11756258Online publication date: 3-Aug-2023
https://doi.org/10.3389/fcomm.2023.1175625
Colley MKränzle TRukzio E(2022)Accessibility-Related Publication Distribution in HCI Based on a Meta-AnalysisExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3519701(1-28)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3519701
Säuberli A(2021)Measuring Text Comprehension for People with Reading Difficulties Using a Mobile ApplicationProceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3441852.3476474(1-4)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3441852.3476474
Alonzo OTrussell JDingman BHuenerfauth MKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Comparison of Methods for Evaluating Complexity of Simplified Texts among Deaf and Hard-of-Hearing Adults at Different Literacy LevelsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445038(1-12)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445038
Nurgalieva LJara Laconich JBaez MCasati FMarchese M(2019)A Systematic Literature Review of Research-Derived Touchscreen Design Guidelines for Older AdultsIEEE Access10.1109/ACCESS.2019.28984677(22035-22058)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2019.2898467
Berke LKafle SHuenerfauth MMandryk RHancock MPerry MCox A(2018)Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy LevelsProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173665(1-12)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173665
Ojha PIsmail AKuppusamy K(2018)Perusal of readability with focus on web content understandabilityJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2018.03.007Online publication date: Mar-2018
https://doi.org/10.1016/j.jksuci.2018.03.007
Chinn DHomeyard C(2016)Easy read and accessible information for people with intellectual disabilities: Is it worth it? A meta‐narrative literature reviewHealth Expectations10.1111/hex.1252020:6(1189-1200)Online publication date: 16-Nov-2016
https://doi.org/10.1111/hex.12520
Nelson EVerhagen TNoordzij M(2016)Health empowerment through activity trackersComputers in Human Behavior10.1016/j.chb.2016.03.06562:C(364-374)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1016/j.chb.2016.03.065
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten