ABSTRACT
Machine Learning (ML) based decision support systems are often like a black box to non-expert users. Here user's confidence becomes critical for effective decision making and maintaining trust in the system. We find that user confidence varies significantly depending on supplementary material presented on screen. We investigate change in user confidence (in the context of ML based decision making) by varying level of uncertainty presented (in an online water-pipe failure prediction case study) and find that all 26 subjects rated higher uncertainty task to be most difficult and had lowest user confidence in predictive decisions of the same. This agrees with our expectation that increased uncertainty would reduce user confidence in predictive decision making. However, ML-researchers subgroup reported being most confident when uncertainty with known probability was presented, whereas other subgroups (viz. general staff and non-ML researchers) appeared most confident when uncertainty was not at all presented. This is an original research to improve understanding of user's decision making confidence with respect to uncertainty presented in machine learning context.
- Allen, P. M., Edwards, J. A., Snyder, F. J., Makinson, K. A., & Hamby, D. M. (2014) The Effect of Cognitive Load on Decision Making with Graphically Displayed Uncertainty Information. Risk Analysis, 34(8), 1495--1505.Google ScholarCross Ref
- Arshad, S., Wang, Y., & Chen, F. (2015). Interactive Mouse Stream as Real-Time Indicator of User's Cognitive Load. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (pp. 1025--1030). ACM. Google ScholarDigital Library
- Camerer, C., & Weber, M., (1992) Recent developments in modeling preferences: Uncertainty and ambiguity. Journal of risk and uncertainty, 5(4), 325--370.Google ScholarCross Ref
- Hsu, M., Bhatt, M., Adolphs, R., Tranel, D., & Camerer, C. F. (2005) Neural systems responding to degrees of uncertainty in human decision-making. Science, 310(5754), 1680--1683.Google ScholarCross Ref
- Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, B. T., & Platt, M. L. (2006) Neural signatures of economic preferences for risk and ambiguity. Neuron, 49(5), 765--775.Google ScholarCross Ref
- Ibrekk, H., & Morgan, M. G. (1992) Graphical communication of uncertain quantities to nontechnical people. Risk analysis, 7(4), 519--529.Google Scholar
- Joslyn, S., & LeClerc, J. (2013) Decisions with Uncertainty: the glass half full. Current Directions in Psychological Science, 22(4), 308--315.Google ScholarCross Ref
- Joslyn, S. L., & Nichols, R. M. (2009). Probability or frequency? Expressing forecast uncertainty in public weather forecasts. Meteorological Applications, 16(3), 309--314.Google ScholarCross Ref
- Kahneman, D., & Tversky, A. (1979) Prospect theory: An analysis of decision under risk. Econometrica, 47, 263--292.Google ScholarCross Ref
- Kiani, R., Corthell, L., & Shadlen, M. N. (2014). Choice certainty is informed by both evidence and decision time. Neuron, 84(6), 1329--1342.Google ScholarCross Ref
- Koriat, A. (2012). The self-consistency model of subjective confidence. Psychological review, 119(1), 80.Google Scholar
- LeClerc, J., and Joslyn, S. (2015) The Cry Wolf Effect and Weather-Related Decision Making. Risk analysis 35, no. 3: 385--395.Google Scholar
- Li, Z., Zhang, B., Wang, Y., Chen, F., Taib, R., Whiffin, V., & Wang, Y. (2014) Water Pipe Condition Assessment: A Hierarchical Beta Process Approach for Sparse Incident Data. Machine Learning, 95(1), 11--26. Google ScholarDigital Library
- Morgado, P., Sousa, N., & Cerqueira, J. J. (2014) The impact of stress in decision making in the context of uncertainty. Journal of Neuroscience Research.Google Scholar
- Moran, R., Teodorescu, A. R., & Usher, M. (2015). Post choice information integration as a causal determinant of confidence: Novel data and a computational account. Cognitive psychology, 78, 99--147.Google Scholar
- Peterson, D. K., & Pitz, G. F. (1988) Confidence, uncertainty, and the use of information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(1), 85.Google ScholarCross Ref
- Pallier, G., Wilkinson, R., Danthiir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002). The role of individual differences in the accuracy of confidence judgments. The Journal of General Psychology, 129(3), 257--299.Google ScholarCross Ref
- Platt, M. L., & Huettel, S. A. (2008) Risky business: the neuroeconomics of decision making under uncertainty. Nature neuroscience, 11(4), 398--403.Google Scholar
- Stanovich, K. E., & Toplak, M. E. (2012). Defining features versus incidental correlates of Type 1 and Type 2 processing. Mind & Society, 11(1), 3--13.Google ScholarCross Ref
- Tversky, A., & Kahneman, D. (1974) Judgment under uncertainty: Heuristics and biases. Science, 185, 1124--1131.Google ScholarCross Ref
- Winkler, R. L. (2015) The Importance of Communicating Uncertainties in Forecasts: Overestimating the Risks from Winter Storm Juno. Risk Analysis, 35: 349--353. doi: 10.1111/risa.12398.Google ScholarCross Ref
- Zhou, J., Sun, J., Chen, F., Wang, Y., Taib, R., Khawaji, A., & Li, Z. (2015). Measurable decision making with GSR and pupillary analysis for intelligent user interface. ACM Transactions on Computer-Human Interaction (ToCHI), 21(6), 33. Google ScholarDigital Library
Index Terms
Investigating User Confidence for Uncertainty Presentation in Predictive Decision Making
Recommendations
Correlation for user confidence in predictive decision making
OzCHI '16: Proceedings of the 28th Australian Conference on Computer-Human InteractionDespite the recognized value of Machine Learning (ML) techniques and high expectation of applying ML techniques within various applications, significant barriers to the widespread adoption and local implementation of ML approaches still exist in the ...
Measurement Uncertainty in Decision-Making: How to Take Reliable Decisions Under Uncertainty
In this paper, the authors examine a common issue concerning the influence of measurement uncertainty on decisions. In fact, in some practical applications, it can be necessary to put in comparison measurement data with thresholds and limits. It occurs ...
Decision making under uncertainty comprising complete ignorance and probability
This paper investigates a model of decision making under uncertainty comprising opposite epistemic states of complete ignorance and probability. In the first part, a new utility theory under complete ignorance is developed that combines Hurwicz-Arrow's ...
Comments