skip to main content
10.1145/2493394.2493415acmconferencesArticle/Chapter ViewAbstractPublication PagesicerConference Proceedingsconference-collections
research-article

Mining data from the AP CS a exam: patterns, non-patterns, and replication failure

Published: 12 August 2013 Publication History

Abstract

To better understand the learning and assessment of introductory computer science, we performed five complementary analyses on over 28 thousand student responses from the 2004 and 2009 Advanced Placement Computer Science (AP CS) A exams. Analyzing the questions from the 1988 AP CS exam, Reges hypothesized that a small set of questions, which were highly correlated with success on the rest of the exam, might be used to assess students' CS ability before taking a CS class. The pattern Reges observed was not replicated on the 2004 and 2009 AP CS A exams. Our hypotheses for why the pattern was not replicated advances Reges' alternative hypothesis regarding the importance of students' mental model of program execution.

References

[1]
Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, W.H. & Krathwohl, D.R. Taxonomy of educational objectives Handbook 1: cognitive domain. Longman Group Ltd., London, 1956)
[2]
Caspersen, M. E., Larsen, K. D., & Bennedsen, J. (2007). Mental models and programming aptitude. ACM SIGCSE, 38.
[3]
College Board (n.d.). AP Computer Science A Course Home Page. Retrieved from http://apcentral.collegeboard.com/
[4]
College Board (n.d.) AP Marine Biology Simulation Case Study. Retrieved from http://apcentral.collegeboard.com/apc/members/courses/teachers_corner/17410.html
[5]
Dehnadi, S., & Bornat, R. (2006). The Camel Has Two Humps. Middlesex University Working Paper, www.cs.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf
[6]
Dweck (2007). Mindset: The new psychology of success. New York, NY: Random House, Inc.
[7]
Kurland, D. M., & Pea, R. D., (1989). Children's mental models of recursive logo programs. Studying the Novice Programmer (E. Soloway & J.C. Spohrer, Eds.) Hillsdale, New Jersey: Lawrence Erlbaum Associates Inc., 315--323.
[8]
Linn, M., C., & Clancy, M. J. (1992). The case for case studies of programming problems. Communications of the ACM. 35(3), 121--132.
[9]
Lam, A. J., Lewis, C. M., Lu, C., Ornstein, I. B., & Wang, D. (2012). Classifying problems to explain patterns of correlation on the 1988 Advanced Placement Computer Science exam. The Journal of Computing Sciences in Colleges. 112--119.
[10]
Lewis, C. M. (2011). Integrating Students' Prior Knowledge into Pedagogy. ICER. Providence, RI. 139--140.
[11]
Lewis, C. M. (2012). Applications of Out-of-Domain Knowledge in Students' Reasoning about Computer Program State. (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses. (Accession Order No. 12710).
[12]
Lewis, C. M., Yasuhara, K., & Anderson, R. E. (2011). Deciding to Major in Computer Science: A Grounded Theory of Students' Self-Assessment of Ability. ICER. Providence, RI. 3--10.
[13]
Lewis, C. M. (2012). The Importance of Students' Attention to Program State: A Case Study of Debugging Behavior. Proceedings of the International Computer Science Education Research Workshop. Auckland, NZ. 127--134.
[14]
Ma, L., Ferguson, J., Roper, M., & Wood, M. (2007). Investigating the viability of mental models held by novice programmers. ACM SIGCSE, 38.
[15]
Reges, S. (2008) They mystery of b := (b = false). ACM SIGCSE, 39, 21--25.
[16]
Robins, A. (2010). Learning edge momentum: a new account of outcomes in CS1. Computer Science Education, 20(1), 37--71.
[17]
Sheard, J., Simon, Carbon, A., Chinn, D., Clear, T., Corney, Malcolm, C., D'Souza, D., Fenwick, J., Harland, J., Laakso, M-J., Teague, D. (2013). How difficult are exams? A framework for assessing the complexity of introductory programming exams. Proceedings of the Fifteenth Australasian Computing Education Conference, Adelaide, Australia, 145--154.
[18]
Shenk, D. (2009). The genius in all of us: New insights into genetics, talent, and IQ. New York, NY: Doubleday.
[19]
Thompson, E., Luxton-Reilly, A., Whalley, J., Hu, M., & Robbins, P. (2008). Bloom's taxonomy for CS assessment. ACE 2008. ICER. Aarhus, Denmark. 69--76.
[20]
Venables, A., Tan, G., & Lister, R. (2009). A closer look at tracing, explaining code writing skills in the novice programmer. ICER. 117--128.

Cited By

View all
  • (2024)Using Benchmarking Infrastructure to Evaluate LLM Performance on CS Concept Inventories: Challenges, Opportunities, and CritiquesProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671097(452-468)Online publication date: 12-Aug-2024
  • (2023)The Impact of Certified CS Teachers on AP Computer Science Exam Scores: A Study in WisconsinProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 210.1145/3568812.3603486(30-31)Online publication date: 7-Aug-2023
  • (2022)Replication of an Evaluation of Teacher Training in the Classification of Programming Exercises Using Bloom’s Taxonomy2022 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON52537.2022.9766707(1839-1848)Online publication date: 28-Mar-2022
  • Show More Cited By

Index Terms

  1. Mining data from the AP CS a exam: patterns, non-patterns, and replication failure

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICER '13: Proceedings of the ninth annual international ACM conference on International computing education research
    August 2013
    202 pages
    ISBN:9781450322430
    DOI:10.1145/2493394
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 August 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. advanced placement computer science
    2. bloom's taxonomy
    3. cs1
    4. difficult topics
    5. mental models of program execution
    6. novice programmers
    7. program state
    8. question difficulty
    9. recursion

    Qualifiers

    • Research-article

    Conference

    ICER '13
    Sponsor:
    ICER '13: International Computing Education Research Conference
    August 12 - 14, 2013
    San California, San Diego, USA

    Acceptance Rates

    ICER '13 Paper Acceptance Rate 22 of 70 submissions, 31%;
    Overall Acceptance Rate 189 of 803 submissions, 24%

    Upcoming Conference

    ICER 2025
    ACM Conference on International Computing Education Research
    August 3 - 6, 2025
    Charlottesville , VA , USA

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Using Benchmarking Infrastructure to Evaluate LLM Performance on CS Concept Inventories: Challenges, Opportunities, and CritiquesProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671097(452-468)Online publication date: 12-Aug-2024
    • (2023)The Impact of Certified CS Teachers on AP Computer Science Exam Scores: A Study in WisconsinProceedings of the 2023 ACM Conference on International Computing Education Research - Volume 210.1145/3568812.3603486(30-31)Online publication date: 7-Aug-2023
    • (2022)Replication of an Evaluation of Teacher Training in the Classification of Programming Exercises Using Bloom’s Taxonomy2022 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON52537.2022.9766707(1839-1848)Online publication date: 28-Mar-2022
    • (2020)A Systematic Literature Review of Studies Related to Mental Activities of Software Developers2020 International Conference Nonlinearity, Information and Robotics (NIR)10.1109/NIR50484.2020.9290239(1-11)Online publication date: 3-Dec-2020
    • (2019)Evaluating Instructors' Classification of Programming Exercises Using the Revised Bloom's TaxonomyProceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education10.1145/3304221.3319748(541-547)Online publication date: 2-Jul-2019
    • (2018)On the Triviality of the Assignment StatementProceedings of the 7th Computer Science Education Research Conference10.1145/3289406.3289413(51-57)Online publication date: 10-Oct-2018
    • (2016)Why Don't Some CS0 Students Succeed?Proceedings of the 47th ACM Technical Symposium on Computing Science Education10.1145/2839509.2844667(317-318)Online publication date: 17-Feb-2016
    • (2014)Measuring demographics and performance in computer science education at a nationwide scale using AP CS dataProceedings of the 45th ACM technical symposium on Computer science education10.1145/2538862.2538918(217-222)Online publication date: 5-Mar-2014

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media