skip to main content
10.1145/2538862.2538976acmconferencesArticle/Chapter ViewAbstractPublication PagessigcseConference Proceedingsconference-collections
research-article

Teaching composition quality at scale: human judgment in the age of autograders

Published: 05 March 2014 Publication History

Abstract

We describe an effort to improve the composition quality of student programs: the property that a program can be understood effectively by another person. As a semester-long component of UC Berkeley's first course for majors, CS 61A, we gave students composition guidelines, scores, and qualitative feedback-all generated manually by a course staff of 10 graders for over 700 students. To facilitate this effort, we created a new online tool that allows instructors to provide feedback efficiently at scale. Our system differs from recently developed alternatives in that it is a branch of an industrial tool originally developed for internal code reviews at Google and used extensively by the open-source community. We found that many of the features designed for industrial applications are well-suited for instructional use as well. We extended the system with permissions controls and comment memories tailored for giving educational feedback.
Using this tool improved the consistency of the feedback we gave to students, the efficiency of generating that feedback, and our ability to communicate that feedback to students. Emphasizing composition throughout the course improved the composition of our students' code. The quality of student programs improved by a statistically significant margin (p<0.01) over those from a previous semester, measured by a blind comparison of student submissions.

References

[1]
H. Abelson and G. Sussman. Structure and Interpretation of Computer Programs. MIT Press McGraw-Hill, Cambridge, Mass. New York, 1985.
[2]
P. Bridge and R. Appleyard. A comparison of electronic and paper-based assignment submission and feedback. British Journal of Educational Technology, 39(4):644--650, 2008.
[3]
J. DeNero and A. Muralidharan. The twitter trends project. In Nifty Assignments Track of SIGCSE, New York, NY, USA, 2013. ACM.
[4]
S. H. Edwards and M. A. Perez-Quinones. Web-CAT: automatically grading programming assignments. In Proceedings of the 13th annual conference on Innovation and technology in computer science education, ITiCSE '08, pages 328--328, New York, NY, USA, 2008. ACM.
[5]
D. Heaney and C. Daly. Mass production of individual feedback. In Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education, ITiCSE '04, pages 117--121, New York, NY, USA, 2004. ACM.
[6]
J. Hollingsworth. Automatic graders for programming classes. Communications of the ACM, 3(10):528--529, Oct. 1960.
[7]
D. Jackson and M. Usher. Grading student programs using assyst. In Proceedings of the twenty-eighth SIGCSE technical symposium on Computer science education, SIGCSE '97, pages 335--339, New York, NY, USA, 1997. ACM.
[8]
D. Jones and B. Jamieson. Three generations of online assignment management. What works and why : reflections on learning with technology : ASCILITE, 1997.
[9]
D. Knuth. Literate Programming. Center for the Study of Language and Information, Stanford, Calif, 1992.
[10]
T. MacWilliam and D. J. Malan. Streamlining grading toward better feedback. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education, ITiCSE '13, pages 147--152, New York, NY, USA, 2013. ACM.
[11]
D. Morris. Automatic grading of student's programming assignments: an interactive process and suite of programs. In Frontiers in Education, 2003. FIE 2003 33rd Annual, volume 3, pages S3F--1--6 vol.3, 2003.
[12]
T. Neller. Pig. In Nifty Assignments Track of SIGCSE, New York, NY, USA, 2010. ACM.

Cited By

View all
  • (2018)Understanding the Effects of Lecturer Intervention on Computer Science Student BehaviourProceedings of the 2017 ITiCSE Conference on Working Group Reports10.1145/3174781.3174787(105-124)Online publication date: 30-Jan-2018
  • (2018)Quantitative Evaluation of Student Engagement in a Large-Scale Introduction to Programming Course using a Cloud-based Automatic Grading System2018 IEEE Frontiers in Education Conference (FIE)10.1109/FIE.2018.8658833(1-5)Online publication date: 3-Oct-2018

Index Terms

  1. Teaching composition quality at scale: human judgment in the age of autograders

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGCSE '14: Proceedings of the 45th ACM technical symposium on Computer science education
    March 2014
    800 pages
    ISBN:9781450326056
    DOI:10.1145/2538862
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 March 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. introductory programming
    2. web-based feedback

    Qualifiers

    • Research-article

    Conference

    SIGCSE '14
    Sponsor:

    Acceptance Rates

    SIGCSE '14 Paper Acceptance Rate 108 of 274 submissions, 39%;
    Overall Acceptance Rate 1,787 of 5,146 submissions, 35%

    Upcoming Conference

    SIGCSE TS 2025
    The 56th ACM Technical Symposium on Computer Science Education
    February 26 - March 1, 2025
    Pittsburgh , PA , USA

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)7
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2018)Understanding the Effects of Lecturer Intervention on Computer Science Student BehaviourProceedings of the 2017 ITiCSE Conference on Working Group Reports10.1145/3174781.3174787(105-124)Online publication date: 30-Jan-2018
    • (2018)Quantitative Evaluation of Student Engagement in a Large-Scale Introduction to Programming Course using a Cloud-based Automatic Grading System2018 IEEE Frontiers in Education Conference (FIE)10.1109/FIE.2018.8658833(1-5)Online publication date: 3-Oct-2018

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media