skip to main content
10.1145/1866029.1866067acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

A conversational interface to web automation

Published: 03 October 2010 Publication History

Abstract

This paper presents CoCo, a system that automates web tasks on a user's behalf through an interactive conversational interface. Given a short command such as "get road conditions for highway 88," CoCo synthesizes a plan to accomplish the task, executes it on the web, extracts an informative response, and returns the result to the user as a snippet of text. A novel aspect of our approach is that we leverage a repository of previously recorded web scripts and the user's personal web browsing history to determine how to complete each requested task. This paper describes the design and implementation of our system, along with the results of a brief user study that evaluates how likely users are to understand what CoCo does for them.

Supplementary Material

SWF File (347s-lau.swf)

References

[1]
}}TurKit, 2009. http://groups.csail.mit.edu/uid/turkit/.
[2]
}}J. F. Allen, N. Chambers, G. Ferguson, L. Galescu, H. Jung, M. D. Swift, and W. Taysom. Plow: A collaborative task learning agent. In AAAI, pages 1514--1519. AAAI Press, 2007.
[3]
}}J. P. Bigham, T. Lau, and J. Nichols. TrailBlazer: Enabling Blind Users to Blaze Trails Through the Web. In IUI '09: Proceedings of the 13th international conference on Intelligent user interfaces, pages 177--186, New York, NY, USA, 2009. ACM.
[4]
}}M. Bolin, M. Webber, P. Rha, T. Wilson, and R. C. Miller. Automation and customization of rendered web pages. In UIST '05: Proceedings of the 18th annual ACM symposium on User interface software and technology, pages 163--172, New York, NY, USA, 2005. ACM.
[5]
}}O. Etzioni and D. Weld. A softbot-based interface to the Internet. Commun. ACM, 37(7):72--76, 1994.
[6]
}}G. Leshed, E. M. Haber, T. Matthews, and T. Lau. CoScripter: Automating & Sharing How-To Knowledge in the Enterprise. In Proceedings of the 26th SIGCHI Conference on Human Factors in Computing Systems (CHI '08), pages 1719--1728, Florence, Italy, 2008.
[7]
}}I. Li, J. W. Nichols, and T. Lau. Here's What I Did: Sharing and Reusing Web Activity with ActionShot. In Proceedings of the 28th SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 2010. To appear.
[8]
}}G. Little, T. A. Lau, A. Cypher, J. Lin, E. M. Haber, and E. Kandogan. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. In CHI '07: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 943--946, New York, NY, USA, 2007. ACM.
[9]
}}J. U. Mahmud, Y. Borodin, and I. V. Ramakrishnan. CSurf: a context-driven non-visual web-browser. In WWW '07: Proceedings of the 16th international conference on World Wide Web, pages 31--40, New York, NY, USA, 2007. ACM.
[10]
}}J. Nichols, Z. Hua, and J. Barton. Highlight: a system for creating and deploying mobile web applications. In UIST '08: Proceedings of the 21st annual ACM symposium on User interface software and technology, pages 249--258, New York, NY, USA, 2008. ACM.
[11]
}}H. Takagi, C. Asakawa, K. Fukuda, and J. Maeda. Accessibility designer: visualizing usability for the blind. In Assets '04: Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility, pages 177--184, New York, NY, USA, 2004. ACM.
[12]
}}W. Wahlster. Smartweb: multimodal web services on the road. In MULTIMEDIA '07: Proceedings of the 15th international conference on Multimedia, pages 16--16, New York, NY, USA, 2007. ACM.

Cited By

View all
  • (2024)Conversation-based hybrid UI for the repertory grid technique: A lab experiment into automation of qualitative surveysInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103227184(103227)Online publication date: Apr-2024
  • (2022)Automatically Generating and Improving Voice Command Interface from Operation Sequences on SmartphonesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517459(1-21)Online publication date: 29-Apr-2022
  • (2022)UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital LibraryProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501855(1-24)Online publication date: 29-Apr-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology
October 2010
476 pages
ISBN:9781450302715
DOI:10.1145/1866029
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 October 2010

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. automation
  2. intelligent assistants
  3. natural language interfaces

Qualifiers

  • Research-article

Conference

UIST '10

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25
The 38th Annual ACM Symposium on User Interface Software and Technology
September 28 - October 1, 2025
Busan , Republic of Korea

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Conversation-based hybrid UI for the repertory grid technique: A lab experiment into automation of qualitative surveysInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103227184(103227)Online publication date: Apr-2024
  • (2022)Automatically Generating and Improving Voice Command Interface from Operation Sequences on SmartphonesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517459(1-21)Online publication date: 29-Apr-2022
  • (2022)UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital LibraryProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501855(1-24)Online publication date: 29-Apr-2022
  • (2022)RLBrowse: Generating Realistic Packet Traces with Reinforcement LearningNOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium10.1109/NOMS54207.2022.9789851(1-6)Online publication date: 25-Apr-2022
  • (2021)Etna: Harvesting Action Graphs from WebsitesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474752(312-331)Online publication date: 10-Oct-2021
  • (2021)Firefox Voice: An Open and Extensible Voice Assistant Built Upon the WebProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445409(1-18)Online publication date: 6-May-2021
  • (2021)GliderProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3462905(1420-1430)Online publication date: 11-Jul-2021
  • (2021)FritzBotInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2021.102699155:COnline publication date: 1-Nov-2021
  • (2020)Conversational Interfaces for a Smart CampusProceedings of the 2020 International Conference on Advanced Visual Interfaces10.1145/3399715.3399914(1-5)Online publication date: 28-Sep-2020
  • (2020)Conversational Web Interaction: Proposal of a Dialog-Based Natural Language Interaction Paradigm for the WebChatbot Research and Design10.1007/978-3-030-39540-7_7(94-110)Online publication date: 19-Jan-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media