research-article

Iris: A Conversational Agent for Complex Tasks

Authors:

Julia Mendelsohn,

Jonathan Bassen,

Michael S. BernsteinAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 473, Pages 1 - 12

https://doi.org/10.1145/3173574.3174047

Published: 21 April 2018 Publication History

Abstract

Today, most conversational agents are limited to simple tasks supported by standalone commands, such as getting directions or scheduling an appointment. To support more complex tasks, agents must be able to generalize from and combine the commands they already understand. This paper presents a new approach to designing conversational agents inspired by linguistic theory, where agents can execute complex requests interactively by combining commands through nested conversations. We demonstrate this approach in Iris, an agent that can perform open-ended data science tasks such as lexical analysis and predictive modeling. To power Iris, we have created a domain-specific language that transforms Python functions into combinable automata and regulates their combinations through a type system. Running a user study to examine the strengths and limitations of our approach, we find that data scientists completed a modeling task 2.6 times faster with Iris than with Jupyter Notebook.

Supplementary Material

MP4 File (pn3875.mp4)

Download
244.68 MB

References

[1]

Adar, E., Dontcheva, M. and Laput, G., CommandSpace: modeling the relationships between tasks, descriptions and features, In Proceedings of the 27th annual ACM symposium on User interface software and technology, ACM, 2014

Digital Library

[2]

Allen, J., Chambers, N., Ferguson, G., Galescu, L., Jung, H., Swift, M. and Taysom, W., Plow: A collaborative task learning agent, 2007

[3]

Anderson, E., The species problem in Iris, In Annals of the Missouri Botanical Garden, 1936

[4]

Berant, J., Chou, A., Frostig, R. and Liang, P., Semantic Parsing on Freebase from Question-Answer Pairs., In EMNLP, 2013

[5]

Bohus, D. and Rudnicky, A., The RavenClaw dialog management framework: Architecture and systems, In Computer Speech&Language, 2009

Digital Library

[6]

Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J. and Monroy-Hernández, A., Calendar. help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop, In CHI, 2017

Digital Library

[7]

Fast, E., McGrath, W., Rajpurkar, P. and Bernstein, M., Augur: Mining Human Behaviors from Fiction to Power Interactive Systems, In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, 2016

Digital Library

[8]

Fast, E., Steffee, D., Wang, L., Brandt, J. and Bernstein, M., Emergent, crowd-scale programming practice in the IDE, In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, ACM, 2014

Digital Library

[9]

Fast, E., Chen, B. and Bernstein, M., Empath: Understanding topic signals in large-scale text, In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, 2016

Digital Library

[10]

Fast, E. and Horvitz, E., Identifying dogmatism in social media: Signals and models, In EMNLP, 2016

[11]

Fast, E. and Bernstein, M., Meta: Enabling Programming Languages to Learn from the Crowd, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ACM, 2016

Digital Library

[12]

Fourney, A., Mann, R. and Terry, M., Query-feature graphs: bridging user vocabulary and system functionality, In Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM, 2011

Digital Library

[13]

Gao, T., Dontcheva, M., Adar, E., Liu, Z. and Karahalios, K., Datatone: Managing ambiguity in natural language interfaces for data visualization, In Proceedings of the 28th Annual ACM Symposium on User Interface Software&Technology, ACM, 2015

Digital Library

[14]

Gee, J., An introduction to discourse analysis: Theory and method, Routledge, 2014

[15]

Guo, P. and Seltzer, M., BURRITO: Wrapping Your Lab Notebook in Computational Infrastructure., In TaPP,

Digital Library

[16]

Hartmann, B., MacDougall, D., Brandt, J. and Klemmer, S., What would other programmers do: suggesting solutions to error messages, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2010

Digital Library

[17]

Hauswald, J., Laurenzano, M., Zhang, Y., Li, C., Rovinski, A., Khurana, A., Dreslinski, R., Mudge, T., Petrucci, V., Tang, L. and Mars, J., Sirius: An Open End-to-End Voice and Vision Personal Assistant and Its Implications for Future Warehouse Scale Computers, In Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), ACM, New York, NY, USA,

Digital Library

[18]

Hutchby, I. and Wooffitt, R., Conversation analysis, Polity, 2008

[19]

John, R., Potti, N. and Patel, J., Ava: From Data to Insights Through Conversations., In CIDR, 2017

[20]

Kandel, S., Paepcke, A., Hellerstein, J. and Heer, J., Wrangler: Interactive visual specification of data transformation scripts, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2011

Digital Library

[21]

Kandel, S., Paepcke, A., Hellerstein, J. and Heer, J., Enterprise data analysis and visualization: An interview study, In IEEE Transactions on Visualization and Computer Graphics, 2012

Digital Library

[22]

Kery, M., Horvath, A. and Myers, B., Variolite: Supporting Exploratory Programming by Data Scientists, In CHI, 2017

Digital Library

[23]

Klemmer, S., Sinha, A., Chen, J., Landay, J., Aboobaker, N. and Wang, A., Suede: A Wizard of Oz Prototyping Tool for Speech User Interfaces, In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, ACM, New York, NY, USA, 2000

Digital Library

[24]

Laput, G., Dontcheva, M., Wilensky, G., Chang, W., Agarwala, A., Linder, J. and Adar, E., Pixeltone: A multimodal interface for image editing, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2013

Digital Library

[25]

Lasecki, W., Wesley, R., Nichols, J., Kulkarni, A., Allen, J. and Bigham, J., Chorus: a crowd-powered conversational assistant, In Proceedings of the 26th annual ACM symposium on User interface software and technology, ACM, 2013

Digital Library

[26]

Lasecki, W., Thiha, P., Zhong, Y., Brady, E. and Bigham, J., Answering visual questions with conversational crowd assistants, In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, ACM, 2013

Digital Library

[27]

Li, T., Azaria, A. and Myers, B., SUGILITE: Creating Multimodal Smartphone Automation by Demonstration, In CHI'17, 2017

Digital Library

[28]

Little, G. and Miller, R., Keyword programming in Java, In Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, ACM, 2007

Digital Library

[29]

Lupkowski, P. and Ginzburg, J., A corpus-based taxonomy of question responses, In IWCS 2013 (International Workshop on Computational Semantics), 2013

[30]

Maes, P., Agents that reduce work and information overload, In CACM, 1994

Digital Library

[31]

Maloney, J., Resnick, M., Rusk, N., Silverman, B. and Eastmond, E., The Scratch Programming Language and Environment, In Trans. Comput. Educ., 2010

Digital Library

[32]

Nass, C. and Brave, S., Wired for speech: How voice activates and advances the human-computer relationship, MIT press Cambridge, MA, 2005

Digital Library

[33]

Ng, V., Supervised noun phrase coreference research: The first fifteen years, In Proceedings of the 48th annual meeting of the association for computational linguistics, Association for Computational Linguistics, 2010

Digital Library

[34]

Patel, K., Bancroft, N., Drucker, S., Fogarty, J., Ko, A. and Landay, J., Gestalt: integrated support for implementation and analysis in machine learning, In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM, 2010

Digital Library

[35]

Pennebaker, J., Francis, M. and Booth, R., Linguistic inquiry and word count: LIWC 2001, In Mahway: Lawrence Erlbaum Associates, 2001

[36]

Porcheron, M., Fischer, J. and Sharples, S., "Do animals have accents?": talking with agents in multi-party conversation, In CHI, 2016

[37]

Reinhart, T., The syntactic domain of anaphora, Massachusetts Institute of Technology, 1976

[38]

Rong, X., Yan, S., Oney, S., Dontcheva, M. and Adar, E., CodeMend: Assisting Interactive Programming with Bimodal Embedding, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ACM, 2016

Digital Library

[39]

Searle, J., Speech acts: An essay in the philosophy of language, Cambridge university press, 1969

[40]

Serban, I., Sordoni, A., Bengio, Y., Courville, A. and Pineau, J., Building end-to-end dialogue systems using generative hierarchical neural network models, In arXiv preprint arXiv:1507.04808, 2015

Digital Library

[41]

Setlur, V., Battersby, S., Tory, M., Gossweiler, R. and Chang, A., Eviza: A Natural Language Interface for Visual Analysis, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ACM, 2016

Digital Library

[42]

Suhm, B., Myers, B. and Waibel, A., Multimodal error correction for speech user interfaces, In ACM transactions on computer-human interaction (TOCHI), 2001

Digital Library

[43]

Sun, M., Chen, Y. and Rudnicky, A., An intelligent assistant for high-level task understanding, In Proceedings of the 21st International Conference on Intelligent User Interfaces, ACM, 2016

Digital Library

[44]

Talbot, J., Lee, B., Kapoor, A. and Tan, D., EnsembleMatrix: Interactive Visualization to Support Machine Learning with Multiple Classifiers, In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 2009

Digital Library

[45]

Wang, S., Liang, P. and Manning, C., Learning Language Games through Interaction, In CoRR, 2016

[46]

Weizenbaum, J., ELIZA--a computer program for the study of natural language communication between man and machine, In Communications of the ACM, 1966

Digital Library

[47]

Winograd, T. and Flores, F., Understanding computers and cognition: A new foundation for design, Intellect Books, 1986

Digital Library

[48]

Winograd, T., A language/action perspective on the design of cooperative work, In Human-Computer Interaction, 1987

Digital Library

[49]

Xu, G. and Lam, M., Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant, 2017

Cited By

Do HBrachman MDugan CJohnson JLauer JRai PPan Q(2024)Grounding with Structure: Exploring Design Variations of Grounded Human-AI Collaboration in a Natural Language InterfaceProceedings of the ACM on Human-Computer Interaction10.1145/36869028:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686902
Ren YClement J(2024)Augmenting Human Teams with Robots in Knowledge Work Settings: Insights from the LiteratureACM Transactions on Human-Robot Interaction10.1145/364988413:2(1-34)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3649884
Joshi ASarwar SVarshney SNag SAgrawal SNaik JSerra ESpezzano F(2024)REAPER: Reasoning based Retrieval Planning for Complex RAG SystemsProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680087(4621-4628)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3680087
Show More Cited By

Index Terms

Iris: A Conversational Agent for Complex Tasks
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Assessment with computer agents that engage in conversational dialogues and trialogues with learners

This article describes conversation-based assessments with computer agents that interact with humans through chat, talking heads, or embodied animated avatars. Some of these agents perform actions, interact with multimedia, hold conversations with ...
Multiagent system for joke generation: Humor and emotions combined in human-agent conversation

In this paper we present an innovative work on a multiagent joking conversational system. In our research so far we have shown that implementing humor into a chatterbot can visibly improve its performance. The results presented in this paper are the ...
Integrating Ontologies and Cognitive Conversational Agents in On2Conv
Multi-Agent Systems
Abstract
Multiagent systems have been successfully used in many domains. Being social, they are expected to communicate with human users in natural language. Nevertheless, the natural interaction between agents and humans is still challenging. Chatbot ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

70
Total Citations
View Citations
1,574
Total Downloads

Downloads (Last 12 months)170
Downloads (Last 6 weeks)17

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Do HBrachman MDugan CJohnson JLauer JRai PPan Q(2024)Grounding with Structure: Exploring Design Variations of Grounded Human-AI Collaboration in a Natural Language InterfaceProceedings of the ACM on Human-Computer Interaction10.1145/36869028:CSCW2(1-27)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686902
Ren YClement J(2024)Augmenting Human Teams with Robots in Knowledge Work Settings: Insights from the LiteratureACM Transactions on Human-Robot Interaction10.1145/364988413:2(1-34)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3649884
Joshi ASarwar SVarshney SNag SAgrawal SNaik JSerra ESpezzano F(2024)REAPER: Reasoning based Retrieval Planning for Complex RAG SystemsProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680087(4621-4628)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3680087
Brachman MEl-Ashry ADugan CGeyer W(2024)How Knowledge Workers Use and Want to Use LLMs in an Enterprise ContextExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650841(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650841
Choi SLee HLee YKim J(2024)VIVID: Human-AI Collaborative Authoring of Vicarious Dialogues from Lecture VideosProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642867(1-26)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642867
Huang YZhou YChen RPan CShu XWeng DWu Y(2024)Interactive Table Synthesis With Natural LanguageIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332912030:9(6130-6145)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2023.3329120
Feng YWang XPan BWong KRen YLiu SYan ZMa YQu HChen W(2024)XNLI: Explaining and Diagnosing NLI-Based Visual Data AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324000330:7(3813-3827)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2023.3240003
Sv SK SS PG S(2024)Democratizing Data Science:Using Language Models for Intuitive Data Insights and Visualizations2024 4th International Conference on Pervasive Computing and Social Networking (ICPCSN)10.1109/ICPCSN62568.2024.00177(1065-1069)Online publication date: 3-May-2024
https://doi.org/10.1109/ICPCSN62568.2024.00177
Jin SAbhyankar S(2024)ChatGrid: Power Grid Visualization Empowered by a Large Language Model2024 IEEE Workshop on Energy Data Visualization (EnergyVis)10.1109/EnergyVis63885.2024.00007(12-17)Online publication date: 13-Oct-2024
https://doi.org/10.1109/EnergyVis63885.2024.00007
Wu HChao CYi ZFu Z(2024)Improving Knowledge Asymmetry in Group Discussions with Smart AssistantsHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76806-4_11(138-150)Online publication date: 17-Dec-2024
https://doi.org/10.1007/978-3-031-76806-4_11
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten