Max L. Wilson
Wendy Mackay
Ed Chi
Michael Bernstein
ABSTRACT
The replication of research findings is a cornerstone of good science. Replication confirms results, strengthens research, and makes sure progress is based on solid foundations. CHI, however, rewards novelty and is focused on new results. As a community, therefore, we do not value, facilitate, or reward replication in research, and often take the significant results of a single user study on 20 users to be true. This panel will address the issues surrounding replication in our community, and discuss: a) how much of our broad diverse discipline is 'science', b) how, if at all, we currently see replication of research in our community, c) whether we should place more emphasis on replication in some form, and d) how that should look in our community. The aim of the panel is to make a proposal to future CHI organizers (2 are on the panel) for how we should facilitate replication in the future.
Supplemental Material
Available for Download
Slides from the presentation
- Capra, R., Marchionini, G., Oh, J.S., Stutzman, F. and Zhang, Y., Effects of structure and interaction style on distinct search tasks. In Proc. JCDL 2007, 442--451. 2007 Google Scholar
Digital Library
- Harman, D.K., The TREC conferences. Morgan Kaufmann Multimedia Information And Systems Series247--256. 1997. Google ScholarDigital Library
- Hearst, M.A., Next generation web search: setting our sites. IEEE Data Engineering Bulletin: Special Issue on Next Generation Web Search, 23(3), 38--48. 2000.Google Scholar
- Jeffries, R., Miller, J.R., Wharton, C. and Uyeda, K., User interface evaluation in the real world: a comparison of four techniques. In Proc. CHI 1991, 119--124. 1991 Google ScholarDigital Library
- Karat, C., Campbell, R. and Fiegel, T., Comparison of empirical testing and walkthrough methods in user interface evaluation. In Comparison of empirical testing and walkthrough methods in user interface evaluation, 397--404. 1992 Google ScholarDigital Library
- Mackay, W.E., Appert, C., Beaudouin-Lafon, M., Chapuis, O., Du, Y., Fekete, J.-D. and Guiard, Y., Touchstone: exploratory design of experiments. In Proc. CHI 2007, 1425--1434. 2007 Google ScholarDigital Library
- schraefel, m.c., Wilson, M.L., Russell, A. and Smith, D.A., mSpace: improving information access to multimedia domains with multimodal exploratory search. Communications of the ACM, 49(4), 47--49. 2006. Google ScholarDigital Library
Index Terms
- RepliCHI - CHI should be replicating and validating results more: discuss
Recommendations
RepliCHI SIG: from a panel to a new submission venue for replication
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing SystemsAt CHI2011 we ran a panel on how the CHI community handles the replicability of research and the reproducibility of findings. Careful scientific scholarship should build on firm foundations, which includes re-examining old evidences in the face of new ...
Quorum-based synchronization protocols for multimedia replicas
Multiple replicas of multimedia objects are distributed to peers in overlay networks. In quorum-based (QB) protocols, every replica may not be up-to-date and the up-to-date replica can be found in the version counter. Multimedia objects are ...
Consistent and automatic replica regeneration
Reducing management costs and improving the availability of large-scale distributed systems require automatic replica regeneration, that is, creating new replicas in response to replica failures. A major challenge to regeneration is maintaining ...
Comments