skip to main content
10.1145/1753326.1753529acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Interactive optimization for steering machine classification

Published: 10 April 2010 Publication History

Abstract

Interest has been growing within HCI on the use of machine learning and reasoning in applications to classify such hidden states as user intentions, based on observations. HCI researchers with these interests typically have little expertise in machine learning and often employ toolkits as relatively fixed "black boxes" for generating statistical classifiers. However, attempts to tailor the performance of classifiers to specific application requirements may require a more sophisticated understanding and custom-tailoring of methods. We present ManiMatrix, a system that provides controls and visualizations that enable system builders to refine the behavior of classification systems in an intuitive manner. With ManiMatrix, users directly refine parameters of a confusion matrix via an interactive cycle of re-classification and visualization. We present the core methods and evaluate the effectiveness of the approach in a user study. Results show that users are able to quickly and effectively modify decision boundaries of classifiers to tai-lor the behavior of classifiers to problems at hand.

References

[1]
20 Newsgroups. http://kdd.ics.uci.edu/databases/20newsgroups/20newsgroups.html.
[2]
Ankerst, M., Elsen, C., Ester, M. and Kriegel, H.-P. Visual classification: an interactive approach to decision tree construction. KDD 1999.
[3]
Becker, B., Kohavi, R. and Sommerfield, D. Visualizing the Simple Bayesian Classifier. Information Visualization in Data Mining and Knowledge Discovery, {ed.} Fayyad, U., Grinstein, G. and Wierse, A. (2001).
[4]
Caragea, D., Cook, D. and Honavar, V.G. Gaining insights into support vector machine pattern classifiers using projection-based tour methods. KDD 2001.
[5]
Cover, T.M. and Thomas, A.J. Elements of Information Theory. Wiley-Interscience, Hoboken, NJ, USA, 2006.
[6]
Dai, J. and Cheng, J. HMM Editor: a visual editing tool for profile hidden Markov models. BMC Genomics 9, Suppl1 (2008).
[7]
Drummond, C. and Holte, R.C. Cost curves: An improved method for visualizing classifier performance. Machine Learning 65, 1 (2006).
[8]
Duda, R.O, Hart, P.E. and Stock, D.G. Pattern Classification. Wiley-Interscience, Hoboken, NJ, USA, 2004.
[9]
Evgeniou, T., Pontil, M. and Elisseeff, A. Leave One Out Error, Stability, and Generalization of Voting Combinations of Classifiers. Machine Learning 55, 1 (2004).
[10]
Fails, J.A. and Olsen, D.R.J. Interactive machine learning. IUI 2003.
[11]
Fei-Fei, L., Fergus, R. and Perona, P. One-Shot Learning of Object Categories. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 4 (2006).
[12]
Fogarty, J., Ko, A., Aung, H., Golden, E., Tang, K. and Hudson, S. Examining task engagement in sensor-based statistical models of human interruptibility. CHI 2005.
[13]
Frank, E. and Hall, M. Visualizing class probability estimators. LNCS 2838, Springer (2003).
[14]
Garner, S.R. WEKA: The Waikato Environment for Knowledge Analysis. New Zealand Computer Science Research Students Conference, (1995).
[15]
Grimes, D., Tan, D., Hudson, S., Shenoy, P. and Rao, R. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. CHI 2008.
[16]
Horvitz, E. and Apacible, J. Learning and Reasoning about Interruption. ICMI 2003.
[17]
Kapoor, A. and Horvitz, E. Experience sampling for building predictive user models: a comparative study. CHI 2008.
[18]
Moustakis, V. Do People in HCI Use Machine Learning? HCI 2, (1997).
[19]
Patel, K., Fogarty, J., Landay, J. and Harrison, B. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning. AAAI 2008.
[20]
Picard, R.W. Affective computing. MIT Press, Cambridge, MA, USA, 1997.
[21]
Platt, J.C. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, MIT Press (1999).
[22]
Rheingans, P. and desJardins, M. Visualizing High-Dimensional Predictive Model Quality. VIS 2000.
[23]
Saponas, S., Tan, S., Morris, D. and Balakrishnan, R. Demonstrating the feasibility of using forearm electro-myography for muscle-computer interfaces. CHI 2008.
[24]
Stiglic, G., Mertik, M., Podgorelec, V. and Kokol, P. Using Visual Interpretation of Small Ensembles in Microarray Analysis. CMBS 2006.
[25]
Talbot, J., Lee, B., Kapoor, A., and Tan, D. Ensemble Matrix: Interactive Visualization to Support Machine Learning with Multiple Classifiers, CHI 2009.
[26]
Urbanek, S. Exploring Statistical Forests. Joint Statistical Meeting 2002.
[27]
Ware, M., Frank, E., Holmes, G., Hall, M. and Witten, I. Interactive machine learning: letting users build classifiers. Int. J .Human-Computer Studies 56, 3 (2001).

Cited By

View all
  • (2024)Image-to-Text Translation for Interactive Image Recognition: A Comparative User Study with Non-expert UsersJournal of Information Processing10.2197/ipsjjip.32.35832(358-368)Online publication date: 2024
  • (2024)ConfusionLens: Focus+Context Visualization Interface for Performance Analysis of Multiclass Image ClassifiersACM Transactions on Interactive Intelligent Systems10.1145/370013915:1(1-21)Online publication date: 17-Oct-2024
  • (2024)Cooperative Multi-Objective Bayesian Design OptimizationACM Transactions on Interactive Intelligent Systems10.1145/365764314:2(1-28)Online publication date: 17-Apr-2024
  • Show More Cited By

Index Terms

  1. Interactive optimization for steering machine classification

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
      April 2010
      2690 pages
      ISBN:9781605589299
      DOI:10.1145/1753326
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 10 April 2010

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. decision theory
      2. interactive machine learning
      3. interactive optimization
      4. visualization

      Qualifiers

      • Research-article

      Conference

      CHI '10
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)52
      • Downloads (Last 6 weeks)3
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Image-to-Text Translation for Interactive Image Recognition: A Comparative User Study with Non-expert UsersJournal of Information Processing10.2197/ipsjjip.32.35832(358-368)Online publication date: 2024
      • (2024)ConfusionLens: Focus+Context Visualization Interface for Performance Analysis of Multiclass Image ClassifiersACM Transactions on Interactive Intelligent Systems10.1145/370013915:1(1-21)Online publication date: 17-Oct-2024
      • (2024)Cooperative Multi-Objective Bayesian Design OptimizationACM Transactions on Interactive Intelligent Systems10.1145/365764314:2(1-28)Online publication date: 17-Apr-2024
      • (2023)Relative Design Acquisition: A Computational Approach for Creating Visual Interfaces to Steer User ChoicesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581028(1-17)Online publication date: 19-Apr-2023
      • (2023)Increasing User Trust in Optimisation through Feedback and InteractionACM Transactions on Computer-Human Interaction10.1145/350346129:5(1-34)Online publication date: 6-Jan-2023
      • (2023)Polyphony: an Interactive Transfer Learning Framework for Single-Cell Data AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320940829:1(591-601)Online publication date: Jan-2023
      • (2022)A Survey on Explainability in Artificial IntelligenceHandbook of Research on Advances in Data Analytics and Complex Communication Networks10.4018/978-1-7998-7685-4.ch004(55-75)Online publication date: 2022
      • (2022)Users' Expectations, Experiences, and Concerns With COVID Alert, an Exposure-Notification AppProceedings of the ACM on Human-Computer Interaction10.1145/35557706:CSCW2(1-33)Online publication date: 11-Nov-2022
      • (2022)Human and Technological Infrastructures of Fact-checkingProceedings of the ACM on Human-Computer Interaction10.1145/35551436:CSCW2(1-36)Online publication date: 11-Nov-2022
      • (2022)Understanding Account Deletion and Relevant Dark Patterns on Social MediaProceedings of the ACM on Human-Computer Interaction10.1145/35551426:CSCW2(1-43)Online publication date: 11-Nov-2022
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media