skip to main content
10.1145/1816123.1816143acmconferencesArticle/Chapter ViewAbstractPublication PagesjcdlConference Proceedingsconference-collections
research-article

Crowdsourcing the assembly of concept hierarchies

Published: 21 June 2010 Publication History

Abstract

The "wisdom of crowds" is accomplishing tasks that are cumbersome for individuals yet cannot be fully automated by means of specialized computer algorithms. One such task is the construction of thesauri and other types of concept hierarchies. Human expert feedback on the relatedness and relative generality of terms, however, can be aggregated to dynamically construct evolving concept hierarchies. The InPhO (Indiana Philosophy Ontology) project bootstraps feedback from volunteer users unskilled in ontology design into a precise representation of a specific domain. The approach combines statistical text processing methods with expert feedback and logic programming to create a dynamic semantic representation of the discipline of philosophy.
In this paper, we show that results of comparable quality can be achieved by leveraging the workforce of crowdsourcing services such as the Amazon Mechanical Turk (AMT). In an extensive empirical study, we compare the feedback obtained from AMT's workers with that from the InPhO volunteer users providing an insight into qualitative differences of the two groups. Furthermore, we present a set of strategies for assessing the quality of different users when gold standards are missing. We finally use these methods to construct a concept hierarchy based on the feedback acquired from AMT workers.

References

[1]
Omar Alonso, Daniel E. Rose, and Benjamin Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2):9--15, 2008.
[2]
Anupriya Ankolekar, Markus Krötzsch, Duc Thanh Tran, and Denny Vrandecic. The two cultures: Mashing up web 2.0 and the semantic web. Journal of Web Semantics, 6(1):70--75, 2008.
[3]
Daren C. Brabham. Crowdsourcing as a model for problem solving: An introduction and cases. The International Journal of Research into New Media Technologies, 14 (1):75--90, 2008.
[4]
John G. Breslin, Alexandre Passant, and Stefan Decker. The Social Semantic Web. Springer-Verlag, 2010.
[5]
Cameron Buckner, Mathias Niepert, and Colin Allen. From encyclopedia to ontology: Toward dynamic representation of the discipline of philosophy. Synthese, 2010 (forthcoming).
[6]
Tom Gruber. Collective knowledge systems: Where the social web meets the semantic web. Journal of Web Semantics, 6(1):4--13, 2008.
[7]
John Horton. The condition of the turking class: Are online employers fair and honest? CoRR, abs/1001.1172, 2010.
[8]
Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. Data quality from crowdsourcing: a study of annotation selection criteria. In HLT '09: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 27--35, Morristown, NJ, USA, 2009. Association for Computational Linguistics.
[9]
Aniket Kittur, Ed H. Chi, and Bongwon Suh. Crowdsourcing user studies with mechanical turk. In CHI '08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 453--456, New York, NY, USA, 2008. ACM.
[10]
Jaron Lanier. Digital maoism: The hazards of the new online collectivism. Edge, 183, March 30th 2006.
[11]
Mathias Niepert, Cameron Buckner, and Colin Allen. A dynamic ontology for a dynamic reference work. In JCDL '07: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, pages 288--297, New York, NY, USA, 2007. ACM.
[12]
Mathias Niepert, Cameron Buckner, and Colin Allen. Answer set programming on expert feedback to populate and extend dynamic ontologies. In Proceedings of the Twenty-First International Florida Artificial Intelligence Research Society Conference, May 15-17, 2008, Coconut Grove, Florida, USA, pages 500--505, 2008.
[13]
Mathias Niepert, Cameron Buckner, and Colin Allen. Working the crowd: Design principles and early lessons from the social-semantic web. In Proceedings of the Workshop on Web 3.0: Merging Semantic Web and Social Web at ACM Hypertext, 2009.
[14]
J. Ross, A. Zaldivar, L. Irani, and B. Tomlinson. Who are the turkers? worker demographics in amazon mechanical turk. Technical report, Department of Informatics, University of California, Irvine, USA, Technical Report SocialCode-2009-01, 2009.
[15]
Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In KDD '08: Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614--622, New York, NY, USA, 2008. ACM.
[16]
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP '08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254--263, Morristown, NJ, USA, 2008. Association for Computational Linguistics.
[17]
Alexander Sorokin and David Forsyth. Utility data annotation with amazon mechanical turk. Computer Vision and Pattern Recognition Workshop, 0:1--8, 2008.
[18]
Fabian M. Suchanek, Milan Vojnovic, and Dinan Gunawardena. Social tags: meaning and suggestions. In CIKM '08: Proceeding of the 17th ACM conference on Information and knowledge mining, pages 223--232, New York, NY, USA, 2008. ACM.
[19]
Alexis de Tocqueville, Henry Reeve, and John C. Spencer. Democracy in America {microform} / by Alexis de Tocqueville, translated by Henry Reeve. With an original preface and notes by John C. Spencer. G. Dearborn & Co., New York, 1838.
[20]
Luis von Ahn and Laura Dabbish. Labeling images with a computer game. In CHI '04: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319--326, New York, NY, USA, 2004. ACM.
[21]
Luis von Ahn, Mihir Kedia, and Manuel Blum. Verbosity: A game for collecting common-sense knowledge. In ACM Conference on Human Factors in Computing Systems, CHI Notes, 2006.
[22]
Jonathan Zittrain. Work the new digital sweatshops. Newsweek, December 8, 2009.

Cited By

View all
  • (2019)Collaborative Ontology Authoring in the Domain of the Holy Quran Knowledge2019 7th Mediterranean Congress of Telecommunications (CMT)10.1109/CMT.2019.8931398(1-3)Online publication date: Oct-2019
  • (2018)Using microtasks to crowdsource DBpedia entity classificationSemantic Web10.3233/SW-1702619:3(337-354)Online publication date: 1-Jan-2018
  • (2016)Crowd-based ontology engineering with the uComp Protégé pluginSemantic Web10.3233/SW-1501817:4(379-398)Online publication date: 27-May-2016
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
JCDL '10: Proceedings of the 10th annual joint conference on Digital libraries
June 2010
424 pages
ISBN:9781450300858
DOI:10.1145/1816123
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

  • IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. crowdsourcing
  2. similarity
  3. thesaurus learning

Qualifiers

  • Research-article

Conference

JCDL10
Sponsor:
JCDL10: Joint Conference on Digital Libraries
June 21 - 25, 2010
Queensland, Gold Coast, Australia

Acceptance Rates

Overall Acceptance Rate 415 of 1,482 submissions, 28%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2019)Collaborative Ontology Authoring in the Domain of the Holy Quran Knowledge2019 7th Mediterranean Congress of Telecommunications (CMT)10.1109/CMT.2019.8931398(1-3)Online publication date: Oct-2019
  • (2018)Using microtasks to crowdsource DBpedia entity classificationSemantic Web10.3233/SW-1702619:3(337-354)Online publication date: 1-Jan-2018
  • (2016)Crowd-based ontology engineering with the uComp Protégé pluginSemantic Web10.3233/SW-1501817:4(379-398)Online publication date: 27-May-2016
  • (2016)A Comparison of Domain Experts and Crowdsourcing Regarding Concept Relevance Evaluation in Ontology LearningMulti-disciplinary Trends in Artificial Intelligence10.1007/978-3-319-49397-8_21(243-254)Online publication date: 10-Nov-2016
  • (2014)Using Crowdsourcing to Investigate Perception of Narrative SimilarityProceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management10.1145/2661829.2661918(321-330)Online publication date: 3-Nov-2014
  • (2014)Assessing the suitability of an honest rating mechanism for the collaborative creation of structured knowledgeWorld Wide Web10.1007/s11280-012-0193-117:1(85-104)Online publication date: 1-Jan-2014
  • (2014)Evaluation on crowdsourcing researchInformation Systems Frontiers10.1007/s10796-012-9350-416:3(417-434)Online publication date: 1-Jul-2014
  • (2014)The uComp Protégé Plugin: Crowdsourcing Enabled Ontology EngineeringKnowledge Engineering and Knowledge Management10.1007/978-3-319-13704-9_14(181-196)Online publication date: 2014
  • (2014)Populating semantic virtual environmentsComputer Animation and Virtual Worlds10.1002/cav.158725:3-4(405-412)Online publication date: 1-May-2014
  • (2013)Crowdsourced Knowledge AcquisitionInternational Journal on Semantic Web & Information Systems10.4018/ijswis.20130701029:3(14-41)Online publication date: 1-Jul-2013
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media