Interactive vision to detect target objects for helper robots

ABSTRACT

An effective human-robot interaction is essential for wide penetration of service robots into the market. Such robots need vision systems to recognize objects. It is, however, difficult to realize vision systems that can work in various conditions. More robust techniques of object recognition and image segmentation are essential. Thus, we have proposed to use the human user's assistance for object recognition through speech. The robot asks a question to which the user can easily answer and whose answer can efficiently reduce the number of candidate objects even if there are occluded objects and/or objects composed of multicolor parts in the scene. It considers the characteristics of features used for object recognition such as the easiness for humans to specify them by word, thus generating a user-friendly and efficient sequence of questions. Experimental results show that the robot can detect target objects by asking the questions generated by the method.

REFERENCES

• 1: Berry, G. A., Pavlovic, V., and Huang, T. S. BattleView: A multimodal HCI research application. In Proceedings of the Workshop on Perceptual User Interfaces, San Francisco, California, USA, pp. 67--70, November 1998.
• 2: Anita Cremers, Object Reference in Task-Oriented Keyboard Dialogues, Multimodal Human-Computer Communication, Systems, Techniques, and Experiments, p.279-293, January 1998
• 3: Ehrenmann, M., Zollner, R., Rogalla, O., and Dillmann, R. Programming service tasks in household environments by human demonstration. In Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication, Berlin, Germany, pp.460--467, September 2002.
• 4: Zaliyana Mohd Hanafiah , Chizu Yamazaki , Akio Nakamura , Yoshinori Kuno, Human-robot speech interface understanding inexplicit utterances using vision, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986054]
• 5: Hans, M., Graf, B., Schraft, R.D. Robotics home assistant Care-O-bot: Past-present-future. In Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication, Berlin, Germany, pp.380--385, September 2002.
• 6: Hossain, M. A., Kurnia, R., Nakamura, A., and Kuno, Y. Color Objects Segmentation for Helper Robot. In Proceedings of the International Conference on Electrical and Computer Engineering, Dhaka, Bangladesh, pp. 206--209, December 2004.
• 7: Inamura, T., Inaba, M., and Inoue, H. Dialogue Control for Task Achievement based on Evaluation of Situational Vagueness and Stochastic Representation of Experiences. In Proceedings of the International Conference on Intelligent Robots and Systems, Sendai, Japan, pp. 2861--2866, 2004.
• 8: Kawaji, T., Okada, K., Inaba, M., Inoue, H. Human Robot Interaction through Integrating Visual Auditory Information with Relaxation Method. In Proceedings of the International Conference on Multisensor Fusion on Integration for Inteligent Systems, Tokyo, Japan, pp 323 -- 328, 2003.
• 9: Kurnia, R., Hossain, M. A., Nakamura, A., and Kuno, Y. Object Recognition through Human-Robot Interaction by Speech. In Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication, Kurashiki, Okayama, Japan, pp.619--62, September 2004.
• 10: McGuire, P. Fritsch, J., Steil, J.J., Roothling, F., Fink, G.A., Wachsmuth, S., Sagerer, G., Ritter, H. Multi-modal Human Machine Communication for Instruction Robot Grasping Tasks. In Proceedings of the IEEE International Workshop on Robots and Human Interactive Communication, Berlin, Germany, pp. 1082--1089, September 2002.
• 11: Shree K. Nayar , Ruud M. Bolle, Reflectance based object recognition, International Journal of Computer Vision, v.17 n.3, p.219-240, March 1996 [doi>10.1007/BF00128232]
• 12: Poddar, I., Sethi, Y., Ozyildiz, E., and Sharma, R. Toward natural gesture/speech HCI: A case study of weather narration. Proceedings of the Workshop on Perceptual User Interfaces, San Francisco, California, USA, pp. 1--6, November 1998.
• 13: Raisamo, R. A multimodal user interface for public information kiosks. In Proceedings of the Workshop on Perceptual User Interfaces, San Francisco, California, USA, pp. 7--12, November 1998.
• 14: Deb Roy , Bernt Schiele , Alex Pentland, Learning Audio-Visual Associations Using Mutual Information, Proceedings of the Integration of Speech and Image Understanding, p.147, September 21-21, 1999
• 15: Takahashi, T., Nakanishi, S., Kuno, Y., and Shirai, Y. Human-robot interface by verbal and nonverbal communication. In Proceedings of the International Conference on Intelligent Robots and Systems, Victoria, Canada, pp.924--929, October 1998.
• 16: Takizawa, M., Makihara, Y., Shimada, N., Miura, J., and Shirai, Y. A Service Robot with Interactive Vision- Objects Recognition using Dialog with User. In Proceedings of the First International Workshop on Language Understanding and Agents for Real World Interaction, Hokkaido, Japan, 2003.
• 17: Terry Winograd, Understanding Natural Language, Academic Press, Inc., Orlando, FL, 1972
• 18: Yoshizaki, M., Kuno, Y., and Nakamura, A. Mutual assistance between speech and vision for human-robot interface. In Proceedings of the International Conference on Intelligent Robots and Systems, EPFL Lausanne, Switzerland, pp.1308--1313, Septem-October 2002.
• 19: Yoshizaki, M., Nakamura, A., and Kuno, Y. Vision-speech system adapting to the user and environment for service robots. In Proceedings of the International Conference on Intelligent Robots and Systems, Las Vegas, Nevada, USA, pp. 1290--1295, October 2003.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Object recognition

General Terms:
Algorithms, Human Factors, Languages, Theory

Keywords:
human robot interaction, interactive object recognition, multimodal interface, object recognition, segmentation