Prototyping an intelligent agent through Wizard of Oz

ABSTRACT

Turvy is a simulated prototype of an instructible agent. The user teaches it by demonstrating actions and pointing at or talking about relevant data. We formalized our assumptions about what could be implemented, then used the Wizard of Oz to flesh out a design and observe users' reactions as they taught several editing tasks. We found: a) all users invent a similar set of commands to teach the agent; b) users learn the agent's language by copying its speech; c) users teach simple tasks with ease and complex ones with reasonable effort; and d) agents cannot expect users to point to or identify critical features without prompting. In conducting this rather complex simulation, we learned some lessons about using the Wizard of Oz to prototype intelligent agents: a) design of the simulation benefits greatly from prior implementation experience; b) the agent's behavior and dialog capabilities must be based on formal models; c) studies of verbal discourse lead directly to an implementable system; d) the designer benefits greatly by becoming the Wizard; and e) qualitative data is more valuable for answering global concerns, while quantitative data validates accounts and answers fine-grained questions.

REFERENCES

• 1: Allen Cypher, EAGER: programming repetitive tasks by example, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.33-39, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108850]
• 2: Nils Dahlbäck , Arne Jönsson , Lars Ahrenberg, Wizard of Oz studies: why and how, Proceedings of the 1st international conference on Intelligent user interfaces, p.193-200, January 04-07, 1993, Orlando, Florida, United States [doi>10.1145/169891.169968]
• 3: John D. Gould , John Conti , Todd Hovanyecz, Composing letters with a simulated listening typewriter, Proceedings of the 1982 conference on Human factors in computing systems, p.367-370, March 15-17, 1982, Gaithersburg, Maryland, United States [doi>10.1145/800049.801813]
• 4: D.C. Halbert (1984) "Programming by example." Xerox PARC Research report OSD-T8402. Palo Alto CA.
• 5: A. G. Hauptmann, Speech and gestures for graphic image manipulation, Proceedings of the SIGCHI conference on Human factors in computing systems: Wings for the mind, p.241-245, March 1989
• 6: W. C. Hill , J. R. Miller, Justified advice: a semi-naturalistic study of advisory strategies, Proceedings of the SIGCHI conference on Human factors in computing systems, p.185-190, May 15-19, 1988, Washington, D.C., United States [doi>10.1145/57167.57198]
• 7: A. Kay (1984) "Computer software." Scientific American, 251(3), pp. 53-59, September.
• 8: Thomas K. Landauer, Psychology as a mother of invention, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.333-335, April 05-09, 1987, Toronto, Ontario, Canada
• 9: R. G. Leiser, Exploiting convergence to improve natural language understanding, Interacting with Computers, v.1 n.3, p.284-298, Dec. 1989 [doi>10.1016/0953-5438(89)90016-7]
• 10: David L. Maulsby , Ian H. Witten , Kenneth A. Kittlitz, Metamouse: specifying graphical procedures by example, Proceedings of the 16th annual conference on Computer graphics and interactive techniques, p.127-136, July 1989
• 11: David Maulsby, The Turvy experience: simulating an instructible interface, Watch what I do: programming by demonstration, MIT Press, Cambridge, MA, 1993
• 12: David L. Maulsby, Prototyping an instructible interface: Moctec, Proceedings of the SIGCHI conference on Human factors in computing systems, p.153-154, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142778]
• 13: Brad A. Myers, Text formatting by demonstration, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.251-256, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108904]
• 14: J. Wilson, D. Rosenberg (1988) "Rapid prototyping for user interface design." In Handbook of Human-Computer Interaction, pp. 859-875, M. Helander ed., New York, North-Holland.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human factors

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Interaction styles (e.g., commands, menus, forms, direct manipulation); Theory and methods

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.1 Applications and Expert Systems
          Subjects: Natural language interfaces

General Terms:
Design, Experimentation, Human Factors, Theory

Keywords:
Wizard of Oz, instructible system, intelligent agent, programming by demonstration, prototyping