Article

Free access

Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction

Authors:

James Fogarty,

Ryan S. Baker,

Scott E. HudsonAuthors Info & Claims

GI '05: Proceedings of Graphics Interface 2005

Pages 129 - 136

Published: 07 May 2005 Publication History

PDF eReader

Abstract

Applications that use sensor-based estimates face a fundamental tradeoff between true positives and false positives when examining the reliability of these estimates, one that is inadequately described by the straightforward notion of accuracy. To address this tradeoff, this paper examines the use of Receiver Operating Characteristic (ROC) curve analysis, a method that has a long history but is under-appreciated in the human computer interaction research community. We present the fundamentals of ROC analysis, the use of the A' statistic to compute the area under an ROC curve, and the equivalence of A' to the Wilcoxon statistic. We then present several case studies, framed in the context of our work on human interruptibility, demonstrating how ROC analysis can yield better results than analyses based on accuracy. These case studies compare sensor-based estimates with human performance, optimize a feature selection process for the area under the ROC curve, and examine end-user selection of a desirable tradeoff.

References

[1]

Bradley, A. P. (1997) The Use of the Area Under the ROC Curve in the Evaluation of Machine Learning Algorithms. Pattern Recognition, 30. 1145--1159.

Digital Library

Google Scholar

[2]

Efron, B. and Tibshirani, R. J. (1993) An Introduction to the Bootstrap. Chapman & Hall, London.

Google Scholar

[3]

Fogarty, J., Hudson, S., Atkeson, C., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J. and Yang, J. (2005) Predicting Human Interruptibility with Sensors. To Appear, ACM Transactions on Computer-Human Interaction (TOCHI).

Digital Library

Google Scholar

[4]

Fogarty, J., Hudson, S. and Lai, J. (2004) Examining the Robustness of Sensor-Based Statistical Models of Human Interruptibility. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2004), 207--214.

Digital Library

Google Scholar

[5]

Fogarty, J., Ko, A. J., Aung, H. H., Golden, E., Tang, K. P. and Hudson, S. E. (2005) Examining Task Engagement in Sensor-Based Statistical Models of Human Interruptibility. To Appear, Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2005).

Digital Library

Google Scholar

[6]

Goffmann, E. On Facework. In Goffmann, E. ed. Interaction Ritual, Random House, New York, 1982, 5--45.

Google Scholar

[7]

Green, D. and Swets, J. Signal Detection Theory and Psychophysics, John Wiley and Sons, New York, 1966, 45--49.

Google Scholar

[8]

Hand, D. J. (1997) Construction and Assessment of Classification Rules. Wiley, Chichester.

Google Scholar

[9]

Hand, D. J. and Till, R. J. (2001) A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning, 45 (2). 171--186.

Digital Library

Google Scholar

[10]

Hanley, J. A. and McNeil, B. J. (1982) The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve. Radiology, 143. 29--36.

Crossref

Google Scholar

[11]

Hudson, S., Fogarty, J., Atkeson, C., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J. and Yang, J. (2003) Predicting Human Interruptibility with Sensors: A Wizard of Oz Feasibility Study. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2003), 257--264.

Digital Library

Google Scholar

[12]

Kohavi, R. and John, G. H. (1997) Wrappers for Feature Subset Selection. Artificial Intelligence, 97 (1-2). 273--324.

Digital Library

Google Scholar

[13]

McFarlane, D. C. (2002) Comparison of Four Primary Methods for Coordinating the Interruption of People in Human-Computer Interaction. Human-Computer Interaction, 17 (1). 63--139.

Digital Library

Google Scholar

[14]

Metz, C. E. (1978) Basic Principles of ROC Analysis. Seminars in Nuclear Medicine, 8 (4). 283--298.

Crossref

Google Scholar

Cited By

View all

Andres-Bray JHutt SBaker RSpikol DViberg OMartínez-Monés AGuo P(2023)Exploring Cross-Country Prediction Model Generalizability in MOOCsProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3593380(183-194)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3593380
Liu QWu JHuang ZWang HNing YChen MChen EYi JZhou B(2023)Federated User Modeling from Hierarchical InformationACM Transactions on Information Systems10.1145/356048541:2(1-33)Online publication date: 3-Apr-2023
https://dl.acm.org/doi/10.1145/3560485
Wang SPhan NWang YZhao YStorey MAdams BHaiduc S(2019)Extracting API tips from developer question and answer websitesProceedings of the 16th International Conference on Mining Software Repositories10.1109/MSR.2019.00058(321-332)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/MSR.2019.00058
Show More Cited By

Recommendations

ROC curves in cost space

ROC curves and cost curves are two popular ways of visualising classifier performance, finding appropriate thresholds according to the operating condition, and deriving useful aggregated measures such as the area under the ROC curve (AUC) or the area ...
Constrained monotone regression of ROC curves and histograms using splines and polynomials
ICIP '95: Proceedings of the 1995 International Conference on Image Processing (Vol.2)-Volume 2 - Volume 2

Receiver operating characteristics (ROC) curves have the property that they start at (0,1) and end at (1,0) and are monotonically decreasing. Furthermore, a parametric representation for the curves is more natural, since ROCs need not be single valued ...
ROC curves and nonrandom data

This paper shows that ROC curves that are constructed with nonrandom data are biased.The magnitude of this bias is explored using simulations.A procedure for plotting consistent ROC curves is introduced.The presented procedure works well with simulated ...

Comments

Information & Contributors

Information

Published In

GI '05: Proceedings of Graphics Interface 2005

May 2005

256 pages

ISBN:1568812655

Program Chairs:
Kori Inkpen
Dalhousie University
,
Michiel van de Panne
The University of British Columbia

Publisher

Canadian Human-Computer Communications Society

Waterloo, Canada

Publication History

Published: 07 May 2005

Author Tags

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 206 of 508 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
1,840
Total Downloads

Downloads (Last 12 months)88
Downloads (Last 6 weeks)15

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Andres-Bray JHutt SBaker RSpikol DViberg OMartínez-Monés AGuo P(2023)Exploring Cross-Country Prediction Model Generalizability in MOOCsProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3593380(183-194)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3593380
Liu QWu JHuang ZWang HNing YChen MChen EYi JZhou B(2023)Federated User Modeling from Hierarchical InformationACM Transactions on Information Systems10.1145/356048541:2(1-33)Online publication date: 3-Apr-2023
https://dl.acm.org/doi/10.1145/3560485
Wang SPhan NWang YZhao YStorey MAdams BHaiduc S(2019)Extracting API tips from developer question and answer websitesProceedings of the 16th International Conference on Mining Software Repositories10.1109/MSR.2019.00058(321-332)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/MSR.2019.00058
Gardner JBrooks CAndres JBaker RLuckin RKlemmer SKoedinger K(2018)Replicating MOOC predictive models at scaleProceedings of the Fifth Annual ACM Conference on Learning at Scale10.1145/3231644.3231656(1-10)Online publication date: 26-Jun-2018
https://dl.acm.org/doi/10.1145/3231644.3231656
Ravari YSpronck PSifa RDrachen AMagerko BRowe J(2017)Predicting victory in a hybrid online competitive gameProceedings of the Thirteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment10.5555/3505326.3505356(207-213)Online publication date: 5-Oct-2017
https://dl.acm.org/doi/10.5555/3505326.3505356
Pelánek RBielikova MHerder EDesmarais MCena F(2017)Measuring Predictive Performance of User ModelsAdjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization10.1145/3099023.3099042(197-201)Online publication date: 9-Jul-2017
https://dl.acm.org/doi/10.1145/3099023.3099042
Daroczy BSiklois DPalovics RBenczur AGangemi ALeonardi SPanconesi A(2015)Text Classification Kernels for Quality Prediction over the C3 Data SetProceedings of the 24th International Conference on World Wide Web10.1145/2740908.2778847(1441-1446)Online publication date: 18-May-2015
https://dl.acm.org/doi/10.1145/2740908.2778847
Molavi Kakhki AKliman-Silver CMislove ASchwabe DAlmeida VGlaser HBaeza-Yates RMoon S(2013)IolausProceedings of the 22nd international conference on World Wide Web10.1145/2488388.2488468(919-930)Online publication date: 13-May-2013
https://dl.acm.org/doi/10.1145/2488388.2488468
Garzó ADaróczy BKiss TSiklósi DBenczúr ASchwabe DAlmeida VGlaser HBaeza-Yates RMoon S(2013)Cross-lingual web spam classificationProceedings of the 22nd International Conference on World Wide Web10.1145/2487788.2488139(1149-1156)Online publication date: 13-May-2013
https://dl.acm.org/doi/10.1145/2487788.2488139
Sao Pedro MBaker RGobert JMontalvo ONakama A(2013)Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skillUser Modeling and User-Adapted Interaction10.1007/s11257-011-9101-023:1(1-39)Online publication date: 1-Mar-2013
https://dl.acm.org/doi/10.1007/s11257-011-9101-0
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

ROC curves in cost space

Constrained monotone regression of ROC curves and histograms using splines and polynomials

ROC curves and nonrandom data