An optimization-based approach to dynamic data content selection in intelligent multimedia interfaces

ABSTRACT

We are building a multimedia conversation system to facilitate information seeking in large and complex data spaces. To provide tailored responses to diverse user queries introduced during a conversation, we automate the generation of a system response. Here we focus on the problem of determining the data content of a response. Specifically, we develop an optimization-based approach to content selection. Compared to existing rule-based or plan-based approaches, our work offers three unique contributions. First, our approach provides a general framework that effectively addresses content selection for various interaction situations by balancing a comprehensive set of constraints (e.g., content quality and quantity constraints). Second, our method is easily extensible, since it uses feature-based metrics to systematically model selection constraints. Third, our method improves selection results by incorporating content organization and media allocation effects, which otherwise are treated separately. Preliminary studies show that our method can handle most of the user situations identified in a Wizard-of-Oz study, and achieves results similar to those produced by human designers.

REFERENCES

• 1: Christopher Ahlberg , Ben Shneiderman, Visual information seeking: tight coupling of dynamic query filters with starfield displays, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.313-317, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191775]
• 2: James Allen , George Ferguson , Amanda Stent, An architecture for more realistic conversational systems, Proceedings of the 6th international conference on Intelligent user interfaces, p.1-8, January 14-17, 2001, Santa Fe, New Mexico, United States [doi>10.1145/359784.359822]
• 3: Elisabeth André , Thomas Rist, The design of illustrated documents as a planning task, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 4: I. Androutsopoulos, G. Ritchie, and P. Thanisch. Natural language interfaces to databases: An introduction. Natural Language Engineering, 1(1):29--81, 1995.
• 5: Y. Arens, E. Hovy, and S. van Mulken. Structure and rules in automated multimedia presentation planning. In Proc. IJCAI, pages 1253--1259, 1993.
• 6: M. Bordegoni , G. Faconti , S. Feiner , M. T. Maybury , T. Rist , S. Ruggieri , P. Trahanias , M. Wilson, A standard reference model for intelligent multimedia presentation systems, Computer Standards & Interfaces, v.18 n.6-7, p.477-496, Dec. 1997 [doi>10.1016/S0920-5489(97)00013-5 ]
• 7: G. Carenini and J. Moore. An empirical study of the influence of user tailoring on evaluative argument effectiveness. In IJCAI '01, pages 1307--1314, 2001.
• 8: Alison Cawsey, Planning interactive explanations, International Journal of Man-Machine Studies, v.38 n.2, p.169-199, Feb. 1993 [doi>10.1006/imms.1993.1009 ]
• 9: Context-Based Multimodal Input Understanding in Conversational Systems, Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p.87, October 14-16, 2002
• 10: Jennifer Chu-Carroll , Sandra Carberry, Collaborative response generation in planning dialogues, Computational Linguistics, v.24 n.3, September 1998
• 11: Thomas H. Cormen , Clifford Stein , Ronald L. Rivest , Charles E. Leiserson, Introduction to Algorithms, McGraw-Hill Higher Education, 2001
• 12: A. Csinger, K. Booth, and D. Poole. AI meets authoring: User models for intelligent multimedia. AI Review, 8(5-6):447--468, 1994.
• 13: Mukesh Dalal , Steven Feiner , Kathleen McKeown , Shimei Pan , Michelle Zhou , Tobias Höllerer , James Shaw , Yong Feng , Jeanne Fromer, Negotiation for automated generation of temporal multimedia presentations, Proceedings of the fourth ACM international conference on Multimedia, p.55-64, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244147]
• 14: Mark Derthick , John Kolojejchick , Steven F. Roth, An interactive visual query environment for exploring data, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.189-198, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263545]
• 15: Steven K. Feiner , Kathleen R. McKeown, Automating the Generation of Coordinated Multimedia Explanations, Computer, v.24 n.10, p.33-41, October 1991 [doi>10.1109/2.97249 ]
• 16: P. Grice. Logic and conversation. In P. Cole and J. Morgan, editors, Syntax and Semantics 3: Speech Acts, pages 41--58. Academic Press, 1975.
• 17: Barbara J. Grosz , Candace L. Sidner, Attention, intentions, and the structure of discourse, Computational Linguistics, v.12 n.3, p.175-204, July-September 1986
• 18: Stephan Kerpedjiev , Giuseppe Carenini , Steven F. Roth , Johanna D. Moore, AutoBrief: a multimedia presentation system for assisting data analysis, Computer Standards & Interfaces, v.18 n.6-7, p.583-593, Dec. 1997 [doi>10.1016/S0920-5489(97)00022-6 ]
• 19: Mark T. Maybury , Wolfgang Wahlster, Readings in intelligent user interfaces, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 20: K. McKeown, S. Feiner, J. Robin, D. Seligmann, and M. Tanenblatt. Generating cross-references for multimedia explanation. In Proc. AAAI '92, pages 12--17, 1992.
• 21: Dan R. Olsen , Jon R. Peachey, Query-by-critique: spoken language access to large lists, Proceedings of the 15th annual ACM symposium on User interface software and technology, October 27-30, 2002, Paris, France [doi>10.1145/571985.572004]
• 22: S. Pan and W. Wen. Designing a speech corpus for instance-based spoken language generation. In Proc. INLG '02, pages 49--56, 2002.
• 23: Steven F. Roth , William E Hefley, Intelligent multimedia presentation systems: research and principles, Intelligent multimedia interfaces, American Association for Artificial Intelligence, Menlo Park, CA, 1993
• 24: Somayajulu G. Sripada , Ehud Reiter , Jim Hunter , Jin Yu, Generating English summaries of time series data using the Gricean maxims, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C. [doi>10.1145/956750.956774]
• 25: Alistair Sutcliffe , Peter Faraday, Designing presentation in multimedia interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.92-98, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191710]
• 26: Kumiko Tanaka-Ishii , Kôiti Hasida , Itsuki Noda, Reactive content selection in the generation of real-time soccer commentary, Proceedings of the 17th international conference on Computational linguistics, p.1282-1288, August 10-14, 1998, Montreal, Quebec, Canada
• 27: M. Walker, S. Whittaker, A. Stent, P. Maloor, J. Moore, M. Johnston, and G. Vasireddy. Speech-Plans: Generating evaluative responses in spoken dialogue. In Proc. INLG '02, pages 73--80, 2002.
• 28: C. Wickens. Processing resources in attention. In R. Pararsuraman and D. Davis, editors, Varieties of Attention. Academic Press, 1983.
• 29: M. Zhou and M. Chen. Automated generation of graphic sketches by examples. In IJCAI '03, pages 65--71, 2003.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Selection process
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

Keywords:
automated generation of multimedia presentations, intelligent multimedia interfaces