Common metrics for human-robot interaction

ABSTRACT

This paper describes an effort to identify common metrics for task-oriented human-robot interaction (HRI). We begin by discussing the need for a toolkit of HRI metrics. We then describe the framework of our work and identify important biasing factors that must be taken into consideration. Finally, we present suggested common metrics for standardization and a case study. Preparation of a larger, more detailed toolkit is in progress.

REFERENCES

• 1: ANSI/AIAA. Guide to Human Performance Measurements, AIAA, Washington, DC, 1993.
• 2: Bajcsy, R. Active perception. Proc. IEEE, 76 (1988), 966--1005.
• 3: Bruce, A., Nourbakhsh, I., and Simmons, R., The role of expressiveness and attention in human-robot interaction. In Proc. AAAI Fall Symposium on Emotional and Intelligent II: The Tangled Knot of Social Cognition, (2001).
• 4: Casper, J., and Murphy, R. Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Transactions on Systems, Man and Cybernetics B, 33 (2003).
• 5: Cassimatis, N., Trafton, G., Schultz, A., Bugajska, M., and Adams, W., A task domain for combining and evaluating robotics and cognitive modeling techniques. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2002).
• 6: Dautenhahn, K., and Werry, I., A quantitative technique for analysing robot-human interactions. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2002).
• 7: Dautenhahn, K., Werry, I., Rae, J., Dickerson, P., Stribling, P., and Odgen, B. Robotic playmates: Analysing interactive competencies of children with autism playing with a mobile robot. In Dautenhahn, K., Bond, A., Canamero, L. and Edmonds, B. eds. Socially Intelligent Agents: Creating Relationships with Computers and Robots, Kluwer Academic Publishers, 2002.
• 8: Draper, J., and Blair, L., Workload, flow and telepresence during teleoperation. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (1996), 1030--1035.
• 9: Draper, J.V., and Kaber, D.B. Human-Robot Interaction. In Mital, A., Ayoub, M.M., Kumar, S., Wang, M.-J. and Landau, K. eds. Industrial and Occupational Ergonomics: Users' Encyclopedia (Encyclopedia of Ergonomics), 1999.
• 10: Drury, J., Scholtz, J., and Yanco, H., Awareness in human-robot interactions. In Proc. IEEE International Conference on Systems, Man and Cybernetics, (2003).
• 11: Endsley, M. Measurement of situation awareness in dynamic systems. Human Factors, 37 (1995), 65--84.
• 12: Fong, T., Nourbakhsh, I., and Dautenhahn, K. A survey of socially interactive robots. Robotics and Autonomous Systems, 42 (2003).
• 13: Terrence Fong , Charles Thorpe, Vehicle Teleoperation Interfaces, Autonomous Robots, v.11 n.1, p.9-18, July 2001 [doi>10.1023/A:1011295826834 ]
• 14: Fong, T., Thorpe, C., and Baur, C. Robot, asker of questions. Robotics and Autonomous Systems, 42 (2003).
• 15: Jennifer Goetz , Sara Kiesler, Cooperation with a robotic assistant, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506492]
• 16: Goldberg, K., and Siegwart, R. Beyond Webcams: Introduction to Online Robots. MIT Press, Cambridge, MA, 2002.
• 17: Goodrich, M., and Olsen, D., Seven principles of efficient human robot interaction. In Proc. IEEE International Conference on Systems, Man and Cybernetics, (2003), 3943--3948.
• 18: Granda, T., Kirkpatrick, M., Julien, T., and Peterson, L., The evolutionary role of humans in the human-robot system. In Proc. Human Factors Society 34th Annual Meeting, (1990), 664--668.
• 19: Gutwin, C., Effects of network delay on group work in shared workspaces. In Proc. European Conferences on Computer Supported Work, (2001).
• 20: Hart, S., and Staveland, L. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In Hancock, P. and Meshkati, N. eds. Human Mental Workload, North-Holland Elsevier Science, 1988.
• 21: Huang, H., and Messina, E., Toward a generic model for the autonomy levels for unmanned systems. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2003).
• 22: Stephen Hughes , Michael Lewis, Robotic camera control for remote exploration, Proceedings of the SIGCHI conference on Human factors in computing systems, p.511-517, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985757]
• 23: Jacoff, A., Messina, E., and Evans, J., A reference test course for autonomous mobile robots. In Proc. SPIE-AeroSense Conference, (Orlando, FL, 2001).
• 24: Jacoff, A., Messina, E., and Evans, J., A standard test course for urban search and rescue robots. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2000).
• 25: Kaber, D., Onal, E., and Endsley, M. Design of automation for telerobots and the effect on performance, operator situation awareness and subjective workload. Human Factors & Ergonomics in Manufacturing, 10 (2000), 409--430.
• 26: Kaber, D.B., and Chow, M.-Y., Human-robot interaction research and an approach to mobile-telerobot interface design. In Proc. XVth Triennial Congress of the International Ergonomics Association (CD-ROM), (Seoul, Korea, 2003).
• 27: Kaber, D.B., Riley, J., Zhou, R., and Draper, J.V., Effects of visual interface design, control interface type, and control latency on performance, telepresence, and workload in a teleoperation task. In Proc. XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Society, (2000), Human Factors and Ergonomics Society, 503--506.
• 28: John D. Lee , Neville Moray, Trust, self-confidence, and operators' adaptation to automation, International Journal of Human-Computer Studies, v.40 n.1, p.153-184, Jan. 1994 [doi>10.1006/ijhc.1994.1007 ]
• 29: Lee, J., and See, K. Trust in automation: designing for appropriate reliance. Human Factors, 46 (2004), 50--80.
• 30: Lewis, M., Wang, J., Manojlovich, J., Hughes, S., and Liu, X., Experiments with attitude: attitude displays for teleoperation. In Proc. IEEE International Conference on Systems, Man, and Cybernetics, (2003).
• 31: Macedo, J., Kaber, D., Endsley, M., Powanusorn, P., and Myung, S. The effects of automated compensation for incongruent axes on teleoperator performance. Human Factors, 40 (1999), 541--553.
• 32: I. Scott MacKenzie , Colin Ware, Lag as a determinant of human performance in interactive systems, Proceedings of the INTERCHI '93 conference on Human factors in computing systems, p.488-493, May 1993, Amsterdam, The Netherlands
• 33: William M. Newman , Michael G. Lamming , Mik Lamming, Interactive System Design, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 34: Nielsen, C.W., Ricks, B., Goodrich, M.A., Bruemmer, D., Few, D., and Walton, M., Snapshots for semantic maps. In Proc. IEEE International Conference on Systems, Man and Cybernetics, (The Hague, The Netherlands, 2004).
• 35: Olsen, D., and Goodrich, M., Metrics for evaluating human-robot interactions. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2003).
• 36: Parasuraman, R. Human use and abuse of automation. In Mouloua, M. and Koonce, J. eds. Human-Automation Interaction, Lawrence Erlbaum Associates, 1997.
• 37: Parasuraman, R., Sheridan, T., and Wickens, C. A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man and Cybernetics B, 30 (2000), 286--297.
• 38: Rantanen, E., and Nunes, A., Taxonomoies of measures in air traffic control research. In Proc. International Symposium on Aviation Psychology, (2003).
• 39: Ricks, B., Nielsen, C.W., and Goodrich, M.A., Ecological displays for robot interaction: A new perspective. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (Sendai, Japan, 2004).
• 40: G. Rodriguez , C. R. Weisbin, A New Method to Evaluate Human-Robot System Performance, Autonomous Robots, v.14 n.2-3, p.165-178, March-May 2003
• 41: Sanders, M., and McCormick, E. Human Factors in Engineering and Design. McGraw-Hill, New York, NY, 1993.
• 42: Jean Scholtz, Theory and Evaluation of Human Robot Interactions, Proceedings of the 36th Annual Hawaii International Conference on System Sciences (HICSS'03) - Track 5, p.125.1, January 06-09, 2003
• 43: Jean Scholtz , Brian Antonishek , Jeff Young, Evaluation of a Human-Robot Interface: Development of a Situational Awareness Methodology, Proceedings of the Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04) - Track 5, p.50130.3, January 05-08, 2004
• 44: Scholtz, J., Antonishek, B., and Young, J., Evaluation of operator interventions in autonomous off-road driving. In Proc. NIST Performance Metrics for Intelligent Systems Workshop, (2003).
• 45: Scholtz, J., Young, J., Drury, J., and Yanco, H., Evaluation of human-robot interaction awareness in search and rescue. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (2004).
• 46: Schulte, J., Rosenberg, C., and Thrun, S., Spontaneous, short-term interaction with mobile robots in public places. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (1999).
• 47: Mohamed A. Sheik-Nainar , David B. Kaber , Mo-Yuen Chow, Control gain adaptation in virtual reality mediated human–telerobot interaction: Research Articles, Human Factors in Ergonomics & Manufacturing, v.15 n.3, p.259-274, May 2005 [doi>10.1002/hfm.v15:3]
• 48: Thomas B. Sheridan, Telerobotics, automation, and human supervisory control, MIT Press, Cambridge, MA, 1992
• 49: Steinfeld, A., Interface lessons for fully and semi-autonomous mobile robots. In Proc. IEEE International Conference on Robotics and Automation (ICRA), (2004).
• 50: Tittle, J., Woods, D., Roesler, A., Howard, M., and Phillips, F., The role of 2-D and 3-D task performance in the design and use of visual displays. In Proc. Human Factors and Ergonomics Society's 46th Annual Meeting, (2002).
• 51: Verma, V. Anecdotes from Rover Field Operations, Unpublished, 2000.
• 52: Wilson, G. Applied use of cardiac and respiration measures: Practical considerations and precautions. Biological Psychology, 34 (1992).
• 53: Wilson, G. Real-time adaptive aiding using psychological operator state assessment. In Harris, D. ed. Engineering Psychology and Cognitive Ergonomics, Ashgate, Aldershot, UK, 2001.
• 54: Yanco, H., and Drury, J., A taxonomy for human-robot interaction. In Proc. AAAI Fall Symposium on Human-Robot Interaction, (2002), 111--119.
• 55: Yanco, H., Drury, J., and Scholtz, J. Beyond usability evaluation: analysis of human-robot interaction at a major robotics competition. Human-Computer Interaction, 19 (2004), 117--149.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.9 Robotics
          Subjects: Operator interfaces

General Terms:
Design, Experimentation, Human Factors, Measurement, Performance, Standardization

Keywords:
human-robot interaction, metrics, unmanned ground vehicles