skip to main content
10.1145/1166253.1166275acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
Article

Translating keyword commands into executable code

Published: 15 October 2006 Publication History

Abstract

Modern applications provide interfaces for scripting, but many users do not know how to write script commands. However, many users are familiar with the idea of entering keywords into a web search engine. Hence, if a user is familiar with the vocabulary of an application domain, we anticipate that they could write a set of keywords expressing a command in that domain. For instance, in the web browsing domain, a user might enter <B>click search button</B>. We call expressions of this form keyword commands, and we present a novel approach for translating keyword commands directly into executable code. Our prototype of this system in the web browsing domain translates <B>click search button</B> into the Chickenfoot code <B>click(findButton("search"))</B>. This code is then executed in the context of a web browser to carry out the effect. We also present an implementation of this system in the domain of Microsoft Word. A user study revealed that subjects could use keyword commands to successfully complete 90% of the web browsing tasks in our study without instructions or training. Conversely, we would expect users to complete close to 0% of the tasks if they had to guess the underlying JavaScript commands with no instructions or training.

Supplementary Material

JPG File (1166275.jpg)
index.html (index.html)
Slides from the presentation
ZIP File (p135-slides.zip)
Supplemental material for Translating keyword commands into executable code
Audio only (1166275.mp3)
Video (1166275.mp4)

References

[1]
Apple Computer. Automator. http://www.apple.com/macosx/features/automator/, accessed June 28, 2006.
[2]
Ballard, B., and Biemann, A. Programming in Natural Language: NLC as a Prototype. ACM/CSC-ER Annual Conference, 228--237. 1979.
[3]
Bruckman, A., Community Support for Constructionist Learning. Computer Supported Cooperative Work, 7(1-2), 47--86, Jan. 1998.
[4]
Bruckman, A., Edwards, E. Should we leverage natural-language knowledge? An analysis of user errors in a natural-language-style programming language. CHI '99, pp. 207--214.
[5]
Burnett, M., Cook, C., and Rothermel, G. End-User Software Engineering. Commun. ACM, 47(9), 53--58, Sept. 2004.
[6]
Bolin, M., Webber, M., Rha, P., Wilson, T., Miller, R. Automation and customization of rendered web pages. UIST 2005, pp. 163--172.
[7]
Cypher, A., Ed. Watch What I Do: Programming by Demonstration. MIT Press, Cambridge, MA, 1993.
[8]
Erwig, M., Abraham, R., Cooperstein, I., and Kollmansberger, S. Automatic generation and maintenance of correct spreadsheets. ICSE 2005, pp. 136--145.
[9]
Katz, B., Felshin, S., Yuret, D., Ibrahim, A., Lin, J., Marton, G., McFarland, A., and Temelkuran, B. Omnibase: Uniform Access to Heterogeneous Data for Question Answering. NLDB 2002, pp. 230-234.
[10]
Pausch, R., et al. Alice: A Rapid Prototyping System for 3D Graphics. IEEE Computer Graphics and Applications, 15(3), 8--11, May 1995.
[11]
Kelleher, C. and Pausch, R., Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Comput. Surv., 37(2), 83--137, 2005.
[12]
Ko, A. J. and Myers, B. A. Designing the Whyline: A Debugging Interface for Asking Why and Why Not Questions. CHI 2004, pp. 151--158.
[13]
Ko, A. J., Myers, B. A., and Aung, H. Six Learning Barriers in End-User Programming Systems. VL/HCC 2004, pp. 199--206.
[14]
Lieberman, H., Ed. Your Wish is My Command: Programming By Example. Morgan Kaufmann, San Francisco, CA, 2001.
[15]
Liu, H., and Lieberman, H., Programmatic Semantics for Natural Language Interfaces. CHI 2005, pp. 1597--1600.
[16]
Miller, L., Natural Language Programming: Styles, Strategies, and Contrasts. IBM Systems Journal, 1981.
[17]
Miller, P., Pane, J., Meter, G., and Vorthmann, S. Evolution of Novice Programming Environments: The Structure Editors of Carnegie Mellon University. Interactive Learning Environments, 4(2), 140--158, 1994.
[18]
Mandelin, D., Xu, L., Bodik, R., and Kimelman, D. Jungloid Mining: Helping to Navigate the API Jungle. PLDI '05, pp. 48--61.
[19]
Myers, B., Pane, J., and Ko, A., Natural Programming Languages and Environments. CACM, 47(9), 47--52, Sept. 2004.
[20]
Phalgune, A., Kissinger, C., Burnett, M., Cook, C., Beckwith, L., and Ruthruff, J. R. Garbage In, Garbage Out? An Empirical Look at Oracle Mistakes by End-User Programmers. VL/HCC 2005, pp. 45--52.
[21]
Porter, M., An algorithm for suffix stripping, Program, 14(3), pp 130--137, 1980.
[22]
Price, D., Riloff E., Zachary J., and Harvey B. NaturalJava: A Natural Language Interface for Programming in Java. IUI 2000, pp. 207--211.
[23]
Sammet, J., The Use of English as a Programming Language. CACM, 9(3), 228--230. 1966.
[24]
Teitelbaum, T. and Reps, T. The Cornell program synthesizer: a syntax-directed programming environment. CACM, 24(9), 563--573, 1981.
[25]
Wiedenbeck, S., Engebretson, A. Comprehension strategies of end-user programmers in an event-driven application. VL/HCC 2004, pp. 207--214.

Cited By

View all
  • (2024)DOĞAL DİL METİNLERİNDEN PROGRAMLAMA DİLİ KODU OLUŞTURMA ÇALIŞMALARI: BİR DERLEME ÇALIŞMASIİstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi10.55071/ticaretfbd.1354040Online publication date: 21-Mar-2024
  • (2024)Evaluation of Code Generation for Simulating Participant Behavior in Experience Sampling Method by Iterative In-Context Learning of a Large Language ModelProceedings of the ACM on Human-Computer Interaction10.1145/36611438:EICS(1-19)Online publication date: 17-Jun-2024
  • (2023)Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory ProgrammingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580919(1-23)Online publication date: 19-Apr-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UIST '06: Proceedings of the 19th annual ACM symposium on User interface software and technology
October 2006
354 pages
ISBN:1595933131
DOI:10.1145/1166253
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2006

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. command languages
  2. end-user programming
  3. natural language processing
  4. web automation

Qualifiers

  • Article

Conference

UIST06

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25
The 38th Annual ACM Symposium on User Interface Software and Technology
September 28 - October 1, 2025
Busan , Republic of Korea

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)10
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)DOĞAL DİL METİNLERİNDEN PROGRAMLAMA DİLİ KODU OLUŞTURMA ÇALIŞMALARI: BİR DERLEME ÇALIŞMASIİstanbul Ticaret Üniversitesi Fen Bilimleri Dergisi10.55071/ticaretfbd.1354040Online publication date: 21-Mar-2024
  • (2024)Evaluation of Code Generation for Simulating Participant Behavior in Experience Sampling Method by Iterative In-Context Learning of a Large Language ModelProceedings of the ACM on Human-Computer Interaction10.1145/36611438:EICS(1-19)Online publication date: 17-Jun-2024
  • (2023)Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory ProgrammingProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580919(1-23)Online publication date: 19-Apr-2023
  • (2021)Natural mapping between voice commands and APIsOpen Computer Science10.1515/comp-2020-012511:1(135-145)Online publication date: 13-Jan-2021
  • (2021)Interactive Explanations: Diagnosis and Repair of Reinforcement Learning Based Agent Behaviors2021 IEEE Conference on Games (CoG)10.1109/CoG52621.2021.9618999(01-08)Online publication date: 17-Aug-2021
  • (2021)Interactive Reinforcement Learning for Autonomous Behavior DesignArtificial Intelligence for Human Computer Interaction: A Modern Approach10.1007/978-3-030-82681-9_11(345-375)Online publication date: 5-Nov-2021
  • (2020)Human Languages in Source CodeProceedings of the Seventh ACM Conference on Learning @ Scale10.1145/3386527.3405916(167-174)Online publication date: 12-Aug-2020
  • (2020)Clustering test steps in natural language toward automating test automationProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3417067(1285-1295)Online publication date: 8-Nov-2020
  • (2019)Comparing Apples and OrangesProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300409(1-12)Online publication date: 2-May-2019
  • (2019)Designing Voice-Controllable APIs2019 IEEE 15th International Scientific Conference on Informatics10.1109/Informatics47936.2019.9119302(000019-000024)Online publication date: Nov-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media