skip to main content
10.1145/1531674.1531692acmconferencesArticle/Chapter ViewAbstractPublication PagesgroupConference Proceedingsconference-collections
research-article

Two peers are better than one: aggregating peer reviews for computing assignments is surprisingly accurate

Published: 10 May 2009 Publication History

Abstract

Scientific peer review, open source software development, wikis, and other domains use distributed review to improve quality of created content by providing feedback to the work's creator. Distributed review is used to assess or improve the quality of a work (e.g., an article). However, it can also provide learning benefits to the participants in the review process. We developed an online review system for beginning computer programming students; it gathers multiple anonymous peer reviews to give students feedback on their programming work. We deployed the system in an introductory programming class and evaluated it in a controlled study. We find that: peer reviews are accurate compared to an accepted evaluation standard, that students prefer reviews from other students with less experience than themselves, and that participating in a peer review process results in better learning outcomes.

References

[1]
Anewalt, K. Using peer review as a vehicle for communication skill development and active learning. J. Comput. Small Coll. 21, 2 (2005), 148--155.
[2]
Bloom, B., Englehart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. Taxonomy of Educational Objectives: The Classification of Educational Goals - Handbook 1: Cognitive Domain.
[3]
David McKay Company, Inc., New York, 1956.
[4]
Cho, K., Chung, T. R., King, W. R., and Schunn, C. Peer-based computer-supported knowledge refinement: an empirical investigation. Commun. ACM 51, 3 (2008), 83--88.
[5]
Clancey, W. J. From guidon to neomycin and heracles in twenty short lessons. AI Mag. 7, 3 (1986), 40--60.
[6]
Collofello, J. S. Teaching technical reviews in a one-semester software engineering course. In SIGCSE '87: Proceedings of the eighteenth SIGCSE technical symposium on Computer science education (New York, NY, USA, 1987), ACM, pp. 222--227.
[7]
Cosley, D., Frankowski, D., Terveen, L., and Riedl, J. Using intelligent task routing and contribution review to help communities build artifacts of lasting value. In CHI '06: Proceedings of the SIGCHI conference on Human Factors in computing systems (New York, NY, USA, 2006), ACM, pp. 1037--1046.
[8]
Denning, T., Kelly, M., Lindquist, D., Malani, R., Griswold, W. G., and Simon, B. Lightweight preliminary peer review: does in-class peer review make sense? In SIGCSE '07: Proceedings of the 38th SIGCSE technical symposium on Computer science education (New York, NY, USA, 2007), ACM, pp. 266--270.
[9]
Foltz, P. W., Laham, D., and Landauer, T. K. Automated essay scoring: Applications to education technology. In Proceedings of ED-MEDIA (1999), pp. 939--944.
[10]
Gehringer, E. Strategies and mechanisms for electronic peer review. Frontiers in Education Conference, 2000. FIE 2000. 30th Annual 1 (2000), F1B/2-F1B/7 vol.1.
[11]
Gehringer, E. F. Electronic peer review and peer grading in computer-science courses. SIGCSE Bull. 33, 1 (2001), 139--143.
[12]
Gehringer, E. F. Electronic peer review builds resources for teaching computer architecture. In Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition (2003), American Society for Engineering Education.
[13]
Gehringer, E. F., Chinn, D. D., Manuel A. Pérez-Qui n., and Ardis, M. A. Using peer review in teaching computing. In SIGCSE '05: Proceedings of the 36th SIGCSE technical symposium on Computer science education (New York, NY, USA, 2005), ACM, pp. 321--322.
[14]
Gehringer, E. F., Ehresman, L. M., and Skrien, D. J. Expertiza: students helping to write an ood text. In OOPSLA '06: Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications (New York, NY, USA, 2006), ACM, pp. 901--906.
[15]
Gotel, O., Scharff, C., and Wildenberg, A. Extending and contributing to an open source web-based system for the assessment of programming problems. In PPPJ '07: Proceedings of the 5th international symposium on Principles and practice of programming in Java (New York, NY, USA, 2007), ACM, pp. 3--12.
[16]
Gotel, O., Scharff, C., and Wildenberg, A. Teaching software quality assurance by encouraging student contributions to an open source web-based system for the assessment of programming assignments. In ITiCSE '08: Proceedings of the 13th annual conference on Innovation and technology in computer science education (New York, NY, USA, 2008), ACM, pp. 214--218.
[17]
Hamer, J., Kell, C., and Spence, F. Peer assessment using aropä. In ACE '07: Proceedings of the ninth Australasian conference on Computing education (Darlinghurst, Australia, Australia, 2007), Australian Computer Society, Inc., pp. 43--54.
[18]
Hamer, J., Ma, K. T. K., and Kwong, H. H. F. A method of automatic grade calibration in peer assessment. In ACE '05: Proceedings of the 7th Australasian conference on Computing education (Darlinghurst, Australia, Australia, 2005), Australian Computer Society, Inc., pp. 67--72.
[19]
Herlocker, J. L., Konstan, J. A., and Riedl, J. Explaining collaborative filtering recommendations. In CSCW '00: Proceedings of the 2000 ACM conference on Computer supported cooperative work (New York, NY, USA, 2000), ACM, pp. 241--250.
[20]
Hinds, P. J. The curse of expertise: The effects of expertise and debiasing methods on predictions of novice performance. Journal of Experimental Psychology: Applied 5, 2 (1999), 205--221.
[21]
Krathwohl, D. R., Bloom, B. S., and Masia, B. Taxonomy of Educational Objectives: The Classification of Educational Goals - Handbook 2: Affective Domain, 1 ed. Longman, London, UK, July 1964.
[22]
Lampe, C., and Resnick, P. Slash(dot) and burn: distributed moderation in a large online conversation space. In CHI '04: Proceedings of the SIGCHI conference on Human factors in computing systems (New York, NY, USA, 2004), ACM, pp. 543--550.
[23]
Liu, E. Z.-F., Lin, S., Chiu, C.-H., and Yuan, S.-M. Web-based peer review: the learner as both adapter and reviewer. Education, IEEE Transactions on 44, 3 (Aug 2001), 246--251.
[24]
Resnick, P., Kuwabara, K., Zeckhauser, R., and Friedman, E. Reputation systems. Commun. ACM 43, 12 (2000), 45--48.
[25]
Silva, E., and Moreira, D. Webcom: a tool to use peer review to improve student interaction. J. Educ. Resour. Comput. 3, 1 (2003), 3.
[26]
Sullivan, S. L. Reciprocal peer reviews. In SIGCSE '94: Proceedings of the twenty-fifth SIGCSE symposium on Computer science education (New York, NY, USA, 1994), ACM, pp. 314--318.
[27]
Topping, K. Peer assessment between students in colleges and universities. Review of Educational Research 68, 3 (1998), 249--276.
[28]
Trivedi, A., Kar, D. C., and Patterson-McNeill, H. Automatic assignment management and peer evaluation. J. Comput. Small Coll. 18, 4 (2003), 30--37.
[29]
Trytten, D. Progressing from small group work to cooperative learning: a case study from computer science. Frontiers in Education Conference, 1999. FIE'99. 29th Annual 2 (1999), 13A4/22--13A4/27 vol.2.
[30]
Trytten, D. A. A design for team peer code review. In SIGCSE '05: Proceedings of the 36th SIGCSE technical symposium on Computer science education (New York, NY, USA, 2005), ACM, pp. 455--459.
[31]
Wolfe, W. J. Online student peer reviews. In CITC5 '04: Proceedings of the 5th conference on Information technology education (New York, NY, USA, 2004), ACM, pp. 33--37.
[32]
Zhang, J., Ackerman, M. S., and Adamic, L. Expertise networks in online communities: structure and algorithms. In WWW '07: Proceedings of the 16th international conference on World Wide Web (New York, NY, USA, 2007), ACM, pp. 221--230.

Cited By

View all
  • (2024)University Students’ Perceptions of Peer Assessment in Oral PresentationsEducation Sciences10.3390/educsci1403022114:3(221)Online publication date: 22-Feb-2024
  • (2024)Improving Grading Fairness and Transparency with Decentralized Collaborative Peer AssessmentProceedings of the ACM on Human-Computer Interaction10.1145/36373508:CSCW1(1-24)Online publication date: 26-Apr-2024
  • (2024)A Systematic Literature Review on Recent Peer Code Review Implementation in Education2024 International Conference on TVET Excellence & Development (ICTeD)10.1109/ICTeD62334.2024.10844661(13-19)Online publication date: 16-Dec-2024
  • Show More Cited By

Index Terms

  1. Two peers are better than one: aggregating peer reviews for computing assignments is surprisingly accurate

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    GROUP '09: Proceedings of the 2009 ACM International Conference on Supporting Group Work
    May 2009
    412 pages
    ISBN:9781605585000
    DOI:10.1145/1531674
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 May 2009

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. collaboration
    2. education
    3. peer review

    Qualifiers

    • Research-article

    Conference

    GROUP09
    Sponsor:
    GROUP09: ACM 2009 International Conference on Supporting Group Work
    May 10 - 13, 2009
    Florida, Sanibel Island, USA

    Acceptance Rates

    GROUP '09 Paper Acceptance Rate 40 of 110 submissions, 36%;
    Overall Acceptance Rate 125 of 405 submissions, 31%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)51
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)University Students’ Perceptions of Peer Assessment in Oral PresentationsEducation Sciences10.3390/educsci1403022114:3(221)Online publication date: 22-Feb-2024
    • (2024)Improving Grading Fairness and Transparency with Decentralized Collaborative Peer AssessmentProceedings of the ACM on Human-Computer Interaction10.1145/36373508:CSCW1(1-24)Online publication date: 26-Apr-2024
    • (2024)A Systematic Literature Review on Recent Peer Code Review Implementation in Education2024 International Conference on TVET Excellence & Development (ICTeD)10.1109/ICTeD62334.2024.10844661(13-19)Online publication date: 16-Dec-2024
    • (2023)How Personality Traits Affect Peer Assessment in Distance LearningTechnology, Knowledge and Learning10.1007/s10758-023-09694-229:1(371-396)Online publication date: 2-Nov-2023
    • (2022)Reflection on peer reviewing as a pedagogical tool in higher educationActive Learning in Higher Education10.1177/1469787421107304524:3(291-303)Online publication date: 13-Jan-2022
    • (2022)Feedback Exchange and Online Affinity: A Case Study of Online Fanfiction WritersProceedings of the ACM on Human-Computer Interaction10.1145/35551276:CSCW2(1-29)Online publication date: 11-Nov-2022
    • (2022)Can Students Review Their Peers?Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 110.1145/3502718.3524762(12-18)Online publication date: 7-Jul-2022
    • (2022)Do Teamwork Guidelines Improve Peer Assessment Accuracy or Attitudes During Collaborative Learning?IEEE Transactions on Education10.1109/TE.2021.313024265:4(485-492)Online publication date: 1-Nov-2022
    • (2022)Developing a system to increase motivation and engagement in student code peer review2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)10.1109/TALE54877.2022.00023(93-98)Online publication date: Dec-2022
    • (2022)Gamification improves the quality of student peer code reviewComputer Science Education10.1080/08993408.2022.212409433:3(458-482)Online publication date: 27-Sep-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media