skip to main content
10.1145/2838739.2838753acmotherconferencesArticle/Chapter ViewAbstractPublication PagesozchiConference Proceedingsconference-collections
research-article

Investigating User Confidence for Uncertainty Presentation in Predictive Decision Making

Authors Info & Claims
Published:07 December 2015Publication History

ABSTRACT

Machine Learning (ML) based decision support systems are often like a black box to non-expert users. Here user's confidence becomes critical for effective decision making and maintaining trust in the system. We find that user confidence varies significantly depending on supplementary material presented on screen. We investigate change in user confidence (in the context of ML based decision making) by varying level of uncertainty presented (in an online water-pipe failure prediction case study) and find that all 26 subjects rated higher uncertainty task to be most difficult and had lowest user confidence in predictive decisions of the same. This agrees with our expectation that increased uncertainty would reduce user confidence in predictive decision making. However, ML-researchers subgroup reported being most confident when uncertainty with known probability was presented, whereas other subgroups (viz. general staff and non-ML researchers) appeared most confident when uncertainty was not at all presented. This is an original research to improve understanding of user's decision making confidence with respect to uncertainty presented in machine learning context.

References

  1. Allen, P. M., Edwards, J. A., Snyder, F. J., Makinson, K. A., & Hamby, D. M. (2014) The Effect of Cognitive Load on Decision Making with Graphically Displayed Uncertainty Information. Risk Analysis, 34(8), 1495--1505.Google ScholarGoogle ScholarCross RefCross Ref
  2. Arshad, S., Wang, Y., & Chen, F. (2015). Interactive Mouse Stream as Real-Time Indicator of User's Cognitive Load. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (pp. 1025--1030). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Camerer, C., & Weber, M., (1992) Recent developments in modeling preferences: Uncertainty and ambiguity. Journal of risk and uncertainty, 5(4), 325--370.Google ScholarGoogle ScholarCross RefCross Ref
  4. Hsu, M., Bhatt, M., Adolphs, R., Tranel, D., & Camerer, C. F. (2005) Neural systems responding to degrees of uncertainty in human decision-making. Science, 310(5754), 1680--1683.Google ScholarGoogle ScholarCross RefCross Ref
  5. Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, B. T., & Platt, M. L. (2006) Neural signatures of economic preferences for risk and ambiguity. Neuron, 49(5), 765--775.Google ScholarGoogle ScholarCross RefCross Ref
  6. Ibrekk, H., & Morgan, M. G. (1992) Graphical communication of uncertain quantities to nontechnical people. Risk analysis, 7(4), 519--529.Google ScholarGoogle Scholar
  7. Joslyn, S., & LeClerc, J. (2013) Decisions with Uncertainty: the glass half full. Current Directions in Psychological Science, 22(4), 308--315.Google ScholarGoogle ScholarCross RefCross Ref
  8. Joslyn, S. L., & Nichols, R. M. (2009). Probability or frequency? Expressing forecast uncertainty in public weather forecasts. Meteorological Applications, 16(3), 309--314.Google ScholarGoogle ScholarCross RefCross Ref
  9. Kahneman, D., & Tversky, A. (1979) Prospect theory: An analysis of decision under risk. Econometrica, 47, 263--292.Google ScholarGoogle ScholarCross RefCross Ref
  10. Kiani, R., Corthell, L., & Shadlen, M. N. (2014). Choice certainty is informed by both evidence and decision time. Neuron, 84(6), 1329--1342.Google ScholarGoogle ScholarCross RefCross Ref
  11. Koriat, A. (2012). The self-consistency model of subjective confidence. Psychological review, 119(1), 80.Google ScholarGoogle Scholar
  12. LeClerc, J., and Joslyn, S. (2015) The Cry Wolf Effect and Weather-Related Decision Making. Risk analysis 35, no. 3: 385--395.Google ScholarGoogle Scholar
  13. Li, Z., Zhang, B., Wang, Y., Chen, F., Taib, R., Whiffin, V., & Wang, Y. (2014) Water Pipe Condition Assessment: A Hierarchical Beta Process Approach for Sparse Incident Data. Machine Learning, 95(1), 11--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Morgado, P., Sousa, N., & Cerqueira, J. J. (2014) The impact of stress in decision making in the context of uncertainty. Journal of Neuroscience Research.Google ScholarGoogle Scholar
  15. Moran, R., Teodorescu, A. R., & Usher, M. (2015). Post choice information integration as a causal determinant of confidence: Novel data and a computational account. Cognitive psychology, 78, 99--147.Google ScholarGoogle Scholar
  16. Peterson, D. K., & Pitz, G. F. (1988) Confidence, uncertainty, and the use of information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(1), 85.Google ScholarGoogle ScholarCross RefCross Ref
  17. Pallier, G., Wilkinson, R., Danthiir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002). The role of individual differences in the accuracy of confidence judgments. The Journal of General Psychology, 129(3), 257--299.Google ScholarGoogle ScholarCross RefCross Ref
  18. Platt, M. L., & Huettel, S. A. (2008) Risky business: the neuroeconomics of decision making under uncertainty. Nature neuroscience, 11(4), 398--403.Google ScholarGoogle Scholar
  19. Stanovich, K. E., & Toplak, M. E. (2012). Defining features versus incidental correlates of Type 1 and Type 2 processing. Mind & Society, 11(1), 3--13.Google ScholarGoogle ScholarCross RefCross Ref
  20. Tversky, A., & Kahneman, D. (1974) Judgment under uncertainty: Heuristics and biases. Science, 185, 1124--1131.Google ScholarGoogle ScholarCross RefCross Ref
  21. Winkler, R. L. (2015) The Importance of Communicating Uncertainties in Forecasts: Overestimating the Risks from Winter Storm Juno. Risk Analysis, 35: 349--353. doi: 10.1111/risa.12398.Google ScholarGoogle ScholarCross RefCross Ref
  22. Zhou, J., Sun, J., Chen, F., Wang, Y., Taib, R., Khawaji, A., & Li, Z. (2015). Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Transactions on Computer-Human Interaction (ToCHI), 21(6), 33. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Investigating User Confidence for Uncertainty Presentation in Predictive Decision Making

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Other conferences
      OzCHI '15: Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction
      December 2015
      691 pages
      ISBN:9781450336734
      DOI:10.1145/2838739

      Copyright © 2015 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 7 December 2015

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed limited

      Acceptance Rates

      OzCHI '15 Paper Acceptance Rate47of97submissions,48%Overall Acceptance Rate362of729submissions,50%

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader