René F. Kizilcec
Andrew J. Saltarelli
ABSTRACT
Despite the fact that anyone can sign up for open online courses, their enrollment patterns reflect the historical underrepresentation of certain sociodemographic groups (e.g. women in STEM disciplines). We theorize that enrollment choices online are shaped by contextual cues that activate stereotypes about numeric representation and climate in brick-and-mortar institutions. A longitudinal matched-pairs experiment with 14 MOOCs (N=29,000) tested this theory by manipulating the presence of a diversity statement on course pages and measuring effects on who enrolls. We found a 3% increase in the proportion of students with lower socioeconomic status. The effect size varied across courses between -0.5 and 7 percentage points. No significant changes in enrollment patterns by gender, age, and national development level occurred. Implications for the use and content of diversity statements and their alternatives are discussed.
- Evan P Apfelbaum, Samuel R Sommers, and Michael I Norton. 2008. Seeing race and seeming racist? Evaluating strategic colorblindness in social interaction. Journal of personality and social psychology 95, 4 (2008), 918.Google Scholar
Cross Ref
- Evan P Apfelbaum, Nicole M Stephens, and Ray E Reagans. 2016. Beyond one-size-fits-all: Tailoring diversity approaches to the representation of social groups. Journal of personality and social psychology 111, 4 (2016), 547.Google ScholarCross Ref
- American Psychological Association et al. 2014. Education and socioeconomic status fact sheet. Retrieved December 30 (2014), 2015.Google Scholar
- Roy F Baumeister and M R Leary. 1995. The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychological Bulletin 117, 3 (may 1995), 497--529.Google ScholarCross Ref
- Eric P Bettinger, Lindsay Fox, Susanna Loeb, and Eric S Taylor. 2017. Virtual classrooms: How online college courses affect student success. American Economic Review 107, 9 (2017), 2855--75.Google ScholarCross Ref
- Jim Blascovich, S. J. Spencer, D. Quinn, and Claude M. Steele. 2001. African Americans and High Blood Pressure: The Role of Stereotype Threat. Psychological Science 12, 3 (2001), 225--229.Google Scholar
- Christopher Brooks, Josh Gardner, and Kaifeng Chen. 2018. How Gender Cues in Educational Video Impact Participation and Retention. In Proceedings of the 13th Intl. Conference of the Learning Sciences. 1835--1842.Google Scholar
- Sorathan Chaturapruek, Thomas Dee, Ramesh Johari, René Kizilcec, and Mitchell Stevens. 2018. How a data-driven course planning tool affects college students' GPA: evidence from two field experiments. In Proceedings of the ACM Conference on Learning at Scale. ACM. Google ScholarDigital Library
- Sapna Cheryan, Andrew N. Meltzoff, and Saenam Kim. 2011. Classrooms matter: The design of virtual classrooms influences gender disparities in computer science classes. Computers & Education 57, 2 (2011), 1825--1835. Google ScholarDigital Library
- Sapna Cheryan, Victoria C Plaut, Paul G Davies, and Claude M. Steele. 2009. Ambient belonging: how stereotypical cues impact gender participation in computer science. Journal of personality and social psychology 97, 6 (2009), 1045--1060.Google ScholarCross Ref
- Isaac Chuang and Andrew Ho. 2016. HarvardX and MITx: Four years of open online courses-fall 2012-summer 2016. (2016). https://ssrn.com/abstract=2889436Google Scholar
- Laurie L Cohen and Janet K Swim. 1995. The differential impact of gender ratios on women and men: Tokenism, self-confidence, and expectations. Personality and Social Psychology Bulletin 21, 9 (1995), 876--884.Google ScholarCross Ref
- Charles D Cowan, Robert M Hauser, Robert A Kominski, Henry M Levin, Samuel R Lucas, Stephen L Morgan, and C Chapman. 2012. Improving the measurement of socioeconomic status for the national assessment of educational progress: A theoretical foundation. Washington: National Center for Education Statistics (2012).Google Scholar
- Paul G Davies, Steven J Spencer, D. M. Quinn, and R. Gerhardstein. 2002. Consuming Images: How Television Commercials that Elicit Stereotype Threat Can Restrain Women Academically and Professionally. Personality and Social Psychology Bulletin 28, 12 (2002), 1615--1628.Google ScholarCross Ref
- Paul G Davies, Steven J Spencer, and Claude M. Steele. 2005. Clearing the air: identity safety moderates the effects of stereotype threat on women's leadership aspirations. Journal of personality and social psychology 88, 2 (2005), 276--287.Google ScholarCross Ref
- D Deming, C Goldin, LF Katz, and N Yuchtman. 2014. Can online learning bend the cost curve of higher education. In American Economic Review.Google Scholar
- Tawanna R Dillahunt, Brian Zengguang Wang, and Stephanie Teasley. 2014. Democratizing higher education: Exploring MOOC use among those who cannot afford a formal education. The International Review of Research in Open and Distributed Learning 15, 5 (2014).Google ScholarCross Ref
- Eric F Dubow, Paul Boxer, and L Rowell Huesmann. 2009. Long-term effects of parentsâĂŹ education on childrenâĂŹs educational and occupational success: Mediation by family interactions, child aggression, and teenage aspirations. Merrill-Palmer quarterly (Wayne State University. Press) 55, 3 (2009), 224.Google Scholar
- Jacquelynne S Eccles and Allan Wigfield. 2002. Motivational beliefs, values, and goals. Annual review of psychology 53, 1 (2002), 109--132.Google Scholar
- Robin J Ely and David A Thomas. 2001. Cultural diversity at work: The effects of diversity perspectives on work group processes and outcomes. Administrative science quarterly 46, 2 (2001), 229--273.Google Scholar
- Maya Escueta, Vincent Quan, Andre Joshua Nickow, and Philip Oreopoulos. 2017. Education Technology: An Evidence-Based Review. (2017), 102.Google Scholar
- Adam D Galinsky, Andrew R Todd, Astrid C Homan, Katherine W Phillips, Evan P Apfelbaum, Stacey J Sasaki, Jennifer A Richeson, Jennifer B Olayon, and William W Maddux. 2015. Maximizing the gains and minimizing the pains of diversity: A policy perspective. Perspectives on Psychological Science 10, 6 (2015), 742--748.Google ScholarCross Ref
- Alan S Gerber and Donald P Green. 2008. Field experiments and natural experiments. In The Oxford Handbook of Political Methodology.Google Scholar
- Scott A Ginder, Janice E Kelly-Reid, and Farrah B Mann. 2017. Enrollment and Employees in Postsecondary Institutions, Fall 2016; and Financial Statistics and Academic Libraries, Fiscal Year 2016: First Look (Provisional Data). NCES 2018-002. National Center for Education Statistics (2017).Google Scholar
- J. D. Hansen and J. Reich. 2015. Democratizingeducation?Examiningaccess and usage patterns in massive open online courses. Science (New York, N.Y.) 350, 6265 (2015), 1245--1248.Google Scholar
- John D Hansen and Justin Reich. 2015. Democratizing education? Examining access and usage patterns in massive open online courses. Science 350, 6265 (2015), 1245--1248.Google Scholar
- Michael J Higgins, Fredrik Sävje, and Jasjeet S Sekhon. 2016. Improving massive experiments with threshold blocking. Proceedings of the National Academy of Sciences 113, 27 (2016), 7369--7376.Google ScholarCross Ref
- Shanna Smith Jaggars. 2014. Democratization of Education for Whom? Online Learning and Educational Equity. Diversity & Democracy 17 (2014). Issue 1. https://www.aacu.org/diversitydemocracy/2014/winter/jaggarsGoogle Scholar
- M. Khan, A. Dougherty, Jr. Lewis, N. A., and D. Sekaquaptewa. 2016. Signaling Diversity: Effects of DepartmentsâĂŹ Diversity Statements on PhD Student Recruitment. In Annual Meeting of the Midwestern Psychological Association, Chicago, IL.Google Scholar
- Rene F. Kizilcec. 2017. Identity, threat, and belonging in online learning environments. Ph.D. Dissertation. Stanford University.Google Scholar
- René F Kizilcec, Glenn M Davis, and Geoffrey L Cohen. 2017. Towards equal opportunities in MOOCs: affirmation reduces gender & social-class achievement gaps in China. In Proceedings of the Fourth (2017) ACM Conference on Learning@Scale. ACM, 121--130. Google ScholarDigital Library
- René F. Kizilcec and Sherif Halawa. 2015. Attrition and Achievement Gaps in Online Learning. In Proceedings of the Second ACM Conference on Learning@Scale. ACM, 57--66. Google ScholarDigital Library
- René F Kizilcec and Andrew J Saltarelli. 2019. Psychologically Inclusive Design. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM. Google ScholarDigital Library
- René F Kizilcec, Andrew J Saltarelli, Justin Reich, and Geoffrey L Cohen. 2017. Closing global achievement gaps in MOOCs. Science (New York, N.Y.) 355, 6322 (2017), 251--252.Google Scholar
- Eric D Knowles, Brian S Lowery, Caitlin M Hogan, and Rosalind M Chow. 2009. On the malleability of ideology: Motivated construals of color blindness. Journal of personality and social psychology 96, 4 (2009), 857.Google ScholarCross Ref
- Laura J Kray, Adam D Galinsky, and Leigh Thompson. 2002. Reversing the Gender Gap in Negotiations: An Exploration of Stereotype Regeneration. Organizational Behavior and Human Decision Processes 87, 2 (2002), 386--409.Google ScholarCross Ref
- Penny Liberatos, Bruce G Link, and Jennifer L Kelsey. 1988. The measurement of social class in epidemiology. Epidemiologic reviews 10, 1 (1988), 87--121.Google Scholar
- Christine Logel, Gregory M Walton, Steven J Spencer, Emma C Iserman, William von Hippel, and Amy E Bell. 2009. Interacting with sexist men triggers social identity threat among female engineers. Journal of personality and social psychology 97, 4 (2009), 578.Google ScholarCross Ref
- Hazel Rose Markus, Claude M Steele, and Dorothy M Steele. 2000. Colorblindness as a barrier to inclusion: Assimilation and nonimmigrant minorities. Daedalus 129, 4 (2000), 233--259.Google Scholar
- Ashley E Martin. 2018. The Divergent Effects of Diversity Ideologies for Race and Gender Relations. Ph.D. Dissertation. Columbia University.Google Scholar
- Ashley E Martin and Katherine W Phillips. 2017. What âĂIJblindnessâĂİ to gender differences helps women see and do: Implications for confidence, agency, and action in male-dominated environments. Organizational Behavior and Human Decision Processes 142 (2017), 28--44.Google ScholarCross Ref
- Allison Master, Sapna Cheryan, Andrew N Meltzoff, Allison Master, Sapna Cheryan, and Andrew N Meltzoff. 2015. Computing Whether She Belongs: Stereotypes Undermine Girls' Interest and Sense of Belonging in Computer Science. Journal of Educational Psychology 107, 3 (2015).Google Scholar
- Patrick F McKay, Derek R Avery, Scott Tonidandel, Mark A Morris, Morela Hernandez, and Michelle R Hebl. 2007. Racial differences in employee retention: Are diversity climate perceptions the key? Personnel psychology 60, 1 (2007), 35--62.Google Scholar
- Rodolfo Mendoza-Denton, Lindsay Shaw-Taylor, Serena Chen, and Eunice Chang. 2009. Ironic effects of explicit gender prejudice on women's test performance. Journal of Experimental Social Psychology 45, 1 (2009), 275--278.Google ScholarCross Ref
- Danaë Metaxa-Kakavouli, Kelly Wang, James A Landay, and Jeff Hancock. 2018. Gender-Inclusive Design: Sense of Belonging and Bias in Web Interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 614. Google ScholarDigital Library
- Mary C Murphy, Claude M. Steele, and James J Gross. 2007. Signaling Threat. Psychological science 18, 10 (2007), 879--885.Google Scholar
- Mary C. Murphy and Valerie Jones Taylor. 2011. The Role of Situational Cues in Signaling and Maintaining Stereotype Threat. In Stereotype Threat: Theory, Process, and Applications, M. Inzlicht and T. Schmader (Eds.). Oxford University Press, New York, NY, USA, Chapter 2, 17--33.Google Scholar
- Mary C Murphy and Gregory M Walton. 2013. From prejudiced people to prejudiced places: A social-contextual approach to prejudice. In Frontiers in Social Psychology Series: Stereotyping and Prejudice., C Stangor and C Crandall (Eds.). Psychology Press, New York, NY, USA.Google Scholar
- Hannah-Hanh D Nguyen and Ann Marie Ryan. 2008. Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence. The Journal of applied psychology 93, 6 (2008), 1314--1334.Google ScholarCross Ref
- Victoria C Plaut. 2010. Diversity science: Why and how difference makes a difference. Psychological Inquiry 21, 2 (2010), 77--99.Google ScholarCross Ref
- Victoria C Plaut,KeciaMThomas,Kyneshawau Hurd,and Celina A Romano. 2018. Do color blindness and multiculturalism remedy or foster discrimination and racism? Current Directions in Psychological Science 27, 3 (2018), 200--206.Google ScholarCross Ref
- Valerie Purdie-Vaughns, Claude M. Steele, Paul G Davies, Ruth Ditlmann, and Jennifer Randall Crosby. 2008. Social identity contingencies: How diversity cues signal threat or safety for African Americans in mainstream institutions. Journal of Personality and Social Psychology 94, 4 (2008), 615--630.Google ScholarCross Ref
- Johnmarshall Reeve, Hyungshim Jang, Pat Hardre, and Mafumi Omura. 2002. Providing a rationale in an autonomy-supportive way as a strategy to motivate others during an uninteresting activity. Motivation and emotion 26, 3 (2002), 183--207.Google Scholar
- Justin Reich and José A Ruipérez-Valiente. 2019. The MOOC pivot. Science 363, 6423 (2019), 130--131.Google Scholar
- Joelle C Ruthig, Raymond P Perry, Nathan C Hall, and Steven Hladkyj. 2004. Optimism and Attributional Retraining: Longitudinal Effects on Academic Achievement, Test Anxiety, and Voluntary Course Withdrawal in College Students 1. Journal of Applied Social Psychology 34, 4 (2004), 709--730.Google ScholarCross Ref
- R M Ryan and E L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. The American psychologist 55, 1 (jan 2000), 68--78.Google Scholar
- Julia E Seaman, I Elaine Allen, and Jeff Seaman. 2018. Grade Increase: Tracking Distance Education in the United States. Babson Survey Research Group (2018).Google Scholar
- Jenessa R Shapiro and Amy M Williams. 2012. The role of stereotype threats in undermining girls' and women's performance and interest in STEM fields. Sex Roles 66, 3-4 (2012), 175--183.Google ScholarCross Ref
- Selcuk R Sirin. 2005. Socioeconomic status and academic achievement: A meta-analytic review of research. Review of educational research 75, 3 (2005), 417--453.Google Scholar
- Steven J. Spencer, Steven Fein, Connie T. Wolfe, Christina Fong, and Meghan A. Dunn. 1998. Automatic Activation of Stereotypes: The Role of Self-Image Threat. Personality and Social Psychology Bulletin 24 (1998), 1139--1152. Issue 11.Google ScholarCross Ref
- Claude M. Steele. 1997. A threat in the air. How stereotypes shape intellectual identity and performance. The American Psychologist 52, 6 (1997), 613--629.Google ScholarCross Ref
- Claude M. Steele, Steven J Spencer, and Joshua Aronson. 2002. Contending with group image: The psychology of stereotype and identity threat. Advances in Experimental Social Psychology 34 (2002), 379--440.Google ScholarCross Ref
- H Tajfel and JC Turner. 1986. The Social Identity Theory of Intergroup Behavior. In Psychology of Intergroup Relations, S. Worchel and WG Austin (Eds.). Nelson-Hall Publishers, Chicago, I.L., 7--24.Google Scholar
- M. Mitchell Waldrop. 2013. Online learning: Campus 2.0. Nature 495, 7440 (2013), 160--163.Google Scholar
- Gregory M Walton and Geoffrey L. Cohen. 2003. Stereotype lift. Journal of Experimental Social Psychology 39, 5 (2003), 456--467.Google ScholarCross Ref
- Gregory M Walton, Christine Logel, Jennifer M Peach, Steven J Spencer, and Mark P Zanna. 2015. Two brief interventions to mitigate a "chilly climate" transform women's experience, relationships, and achievement in engineering. Journal of Educational Psychology 107, 2 (2015), 468--485.Google ScholarCross Ref
- Jeffery L Wilson, Katrina A Meyer, and Larry McNeal. 2012. Mission and diversity statements: What they do and do not say. Innovative Higher Education 37, 2 (2012), 125--139.Google ScholarCross Ref
- Leigh S Wilton, Jessica J Good, Corinne A Moss-Racusin, and Diana T Sanchez. 2015. Communicating more than diversity: The effect of institutional diversity statements on expectations and performance as a function of race and gender. Cultural Diversity and Ethnic Minority Psychology 21, 3 (2015), 315.Google ScholarCross Ref
Index Terms
- Can a diversity statement increase diversity in MOOCs?
Recommendations
Psychologically Inclusive Design: Cues Impact Women's Participation in STEM Education
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsVisual and verbal cues can reinforce barriers to access for women in science, technology, engineering, and math (STEM) disciplines. Psychologically inclusive design is an evidence-based approach to reduce psychological barriers by strategically placing ...
Equality, Diversity, and Inclusion in the CS Curriculum
UKICER '23: Proceedings of the 2023 Conference on United Kingdom & Ireland Computing Education ResearchThe computing discipline has long-standing issues in equality, diversity and inclusion (EDI). Here we reflect on a curriculum review, embedding diversity-related learning outcomes to meet accreditation requirements by Professional, Statutory and ...
Promoting diversity in computing
ITiCSE 2018: Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science EducationIn this paper we present a pilot program at San Francisco State University, Promoting INclusivity in Computing (PINC), that is designed to achieve two goals simultaneously: (i) improving diversity in computing, and (ii) increasing computing literacy in ...
Reviews
Ernest L Hughes
Recently, the dean of a university business school asked me to write a diversity statement as part of an application for a teaching position. I asked him what goes in a diversity statement; he didn't have an answer, and I didn't apply. On the other hand, Kizilcec and Saltarelli do have an answer, at least in part. In a nutshell, focus on the experience of students at the virtual classroom door, so to speak, and "nudge" them inside [1]. This paper presents the results of a field experiment to increase enrollment of students in massive open online courses (MOOCs) from four underrepresented populations by placing a concise diversity statement and graphic on the enrollment pages for courses. Can a diversity statement increase diversity in MOOCs That is to say, can it increase enrollments from diverse and underrepresented populations Yes, it can, according to this research, for MOOC students of lower socioeconomic status (SES). Women No. Older people No. People in less-developed countries No. As a former retailer, these findings reminded me of the "last mile" challenge in (online) retail to get shoppers to choose a product [2]. The paper showcases a meticulous research design and concludes with a comprehensive list of 68 references. As an educator, I have used nudges for retention and completion for individual students, but have pondered what to do for "shop and drop" students who start and withdraw within the first week. An inclusive welcome statement may be part of the answer. Researchers interested in hearing more about Saltarelli's critical thinking and design thinking for inclusive and welcoming online learning experiences are referred to [3]. An introduction to Kizilcec's Future of Learning lab can be found online (http://learning.cis.cornell.edu/). Practitioners curious about inclusion in organizations-the next door after education-should see [4]. Now I know what to write in my diversity statement for the dean.
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments