skip to main content
10.1145/1228716.1228722acmconferencesArticle/Chapter ViewAbstractPublication PageshriConference Proceedingsconference-collections
Article

Developing performance metrics for the supervisory control of multiple robots

Published: 10 March 2007 Publication History

Abstract

Efforts are underway to make it possible for a single operator to effectively control multiple robots. In these high workload situations, many questions arise including how many robots should be in the team (Fan-out), what level of autonomy should the robots have, and when should this level of autonomy change (i.e., dynamic autonomy). We propose that a set of metric classes should be identified that can adequately answer these questions. Toward this end, we present a potential set of metric classes for human-robot teams consisting of a single human operator and multiple robots. To test the usefulness and appropriateness of this set of metric classes, we conducted a user study with simulated robots. Using the data obtained from this study, we explore the ability of this set of metric classes to answer these questions.

References

[1]
M. H. Ashcraft. Cognition. Prentice Hall, third edition, 2002.
[2]
J. Brookshire, S. Singh, and R. Simmons. Preliminary results in sliding autonomy for assembly by coordinated teams. In Proceedings of the International Conference on Robots and Systems, 2004.
[3]
J. W. Crandall, M. A. Goodrich, D. R. O. Jr., and C. W. Nielsen. Validating human-robot systems in multi-tasking environments. IEEE Transactions on Systems, Man, and Cybernetics -- Part A: Systems and Humans, 35(4):438--449, 2005.
[4]
M. L. Cummings and S. Guerlain. An interactive decision support tool for real-time in-flight replanning of autonomous vehicles. In AIAA 3rd "Unmanned Unlimited" Technical Conference, Workshop and Exhibit, 2004.
[5]
M. L. Cummings, C. Nehme, and J. W. Crandall. Predicting operator capacity for supervisory control of multiple UAVs. Innovations in Intelligent UAVs: Theory and Applications, Ed. L. Jain, 2006. In press.
[6]
S. R. Dixon and C. D. Wickens. Unmanned aerial vehicle flight control: False alarms versus misses. In Proceedings of the 12th International Symposium on Aviation Psychology, 2003.
[7]
S. R. Dixon, C. D. Wickens, and D. Chang. Unmanned aerial vehicle ight control: False alarms versus misses. In Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting, 2004.
[8]
J. Drury, J. Scholtz, and H. A. Yanco. Awareness in human-robot interactions. In Proceedings of the IEEE Conference on Systems, Man and Cybernetics, Washington, DC, 2003.
[9]
M. R. Endsley. Design and evaluation for situation awareness enhancement. In Proceedings of the Human Factors Society 32nd Annual Meeting, Santa Monica, CA, 1988.
[10]
M. A. Goodrich, T. W. McLain, J. W. Crandall, J. Johansen, J. Anderson, and J. Sun. Managing autonomy in robot teams: Observations from four experiments. In Proceedings of the 2nd Annual Conference on Human-Robot Interaction, 2007.
[11]
M. A. Goodrich and D. R. Olsen. Seven principles of efficient human robot interaction. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, pages 3943--3948, Washington, DC, 2003.
[12]
M. A. Goodrich, M. Quigley, and K. Cosenzo. Task switching and multi-robot teams. In Proceedings of the Third International Multi-Robot Systems Workshop, 2005.
[13]
T. C. Hankins and G. F. Wilson. A comparison of heart rate, eye activity, eeg and subjective measures of pilot mental workload during ight. Aviation, Space and Environmental Medicine, 69(4):360--367, 1998.
[14]
P. J. Mitchell, M. L. Cummings, and T. B. Sheridan. Mitigation of human supervisory control wait times through automation strategies. Technical report, Humans and Automation Laboratory, Massachusetts Institute of Technology, June 2003.
[15]
D. R. Olsen and M. A. Goodrich. Metrics for evaluating human-robot interactions. In NIST's Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MA, 2003.
[16]
D. R. Olsen and S. B. Wood. Fan-out: Measuring human control of multiple robots. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), 2004.
[17]
R. Parasuraman, S. Galster, P. Squire, H. Furukawa, and C. Miller. A exible delegation-type interfface enhances system performance in human supervison of multiple robots: Empirical studies with roboflag. IEEE Transactions on Systems, Man, and Cybernetics -- Part A: Systems and Humans, 35(4):481--493, 2005.
[18]
R. Parasuraman, T. B. Sheridan, and C. D. Wickens. A model of types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics -- Part A: Systems and Humans, 30(3):286--297, 2000.
[19]
G. Rodriguez and C. R. Weisbin. A new method to evaluate human-robot system performance. Autonomous Robots, 14(2-3):165--178, 2003.
[20]
H. A. Ruff, G. L. Calhoun, M. H. Draper, J. V. Fontejon, and B. J. Guilfoos. Exploring automation issues in supervisory control of multiple uavs. In Proceedings of the Human Performance, Situation Awareness, and Automation Technology Conference, pages 218--222, 2004.
[21]
T. B. Sheridan. Telerobotics, Automation, and Human Supervisory Control. The MIT Press, 1992.
[22]
T. B. Sheridan and W. L. Verplank. Human and computer control of undersea teleoperators. Technical report, Man-Machine Laboratory, Massachusetts Institute of Technology, Cambridge, MA, 1978.
[23]
P. Squire, G. Trafton, and R. Parasuraman. Human control of multiple unmanned vehicles: effects of interface type on execution and task switching times. In Proceeding of the 1st Annual Conference on Human-robot Interaction, pages 26--32, New York, NY, USA, 2006. ACM Press.
[24]
A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz, and M. Goodrich. Common metrics for human-robot interaction. In Proceedings of the 1st Annual Conference on Human-Robot Interaction, 2006.
[25]
J. A. Veltman and A. W. K. Gaillard. Physiological workload reactions to increasing levels of task difficulty. Ergonomics, 41(5):656--669, 1998.
[26]
E. D. Visser, R. Parasuraman, A. Freedy, E. Freedy, and G. Weltman. A comprehensive methodology for assessing human-robot team performance for use in training and simulation. In Proceeding of the Human Factors and Ergonomics Society 50th Annual Meeting, 2006.
[27]
C. Wickens and J. G. Hollands. Engineering Psychology and Human Performance. Prentice Hall, Upper Saddle River, NJ, third edition, 2000.

Cited By

View all
  • (2024)Mobile Robot for Security Applications in Remotely Operated Advanced ReactorsApplied Sciences10.3390/app1406255214:6(2552)Online publication date: 18-Mar-2024
  • (2021)Connecting Human-Robot Interaction and Data VisualizationProceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3434073.3444683(281-292)Online publication date: 8-Mar-2021
  • (2021)Designing Interface Aids to Assist Collaborative Robot Operators in Attention Management2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)10.1109/RO-MAN50785.2021.9515519(264-271)Online publication date: 8-Aug-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
HRI '07: Proceedings of the ACM/IEEE international conference on Human-robot interaction
March 2007
392 pages
ISBN:9781595936172
DOI:10.1145/1228716
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 March 2007

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. fan-out
  2. multi-robot teams
  3. supervisory control

Qualifiers

  • Article

Conference

HRI07
HRI07: International Conference on Human Robot Interaction
March 10 - 12, 2007
Virginia, Arlington, USA

Acceptance Rates

HRI '07 Paper Acceptance Rate 22 of 101 submissions, 22%;
Overall Acceptance Rate 268 of 1,124 submissions, 24%

Upcoming Conference

HRI '25
ACM/IEEE International Conference on Human-Robot Interaction
March 4 - 6, 2025
Melbourne , VIC , Australia

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)27
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Mobile Robot for Security Applications in Remotely Operated Advanced ReactorsApplied Sciences10.3390/app1406255214:6(2552)Online publication date: 18-Mar-2024
  • (2021)Connecting Human-Robot Interaction and Data VisualizationProceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3434073.3444683(281-292)Online publication date: 8-Mar-2021
  • (2021)Designing Interface Aids to Assist Collaborative Robot Operators in Attention Management2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)10.1109/RO-MAN50785.2021.9515519(264-271)Online publication date: 8-Aug-2021
  • (2021)XRTI: eXtended Reality Based Telepresence Interface for Multiple Robot SupervisionIntelligent Human Computer Interaction10.1007/978-3-030-98404-5_20(205-217)Online publication date: 20-Dec-2021
  • (2021)Contextual Evaluation of Human–Machine Team EffectivenessSystems Engineering and Artificial Intelligence10.1007/978-3-030-77283-3_14(283-307)Online publication date: 2-Nov-2021
  • (2020)Coactive design of explainable agent-based task planning and deep reinforcement learning for human-UAVs teamworkChinese Journal of Aeronautics10.1016/j.cja.2020.05.00133:11(2930-2945)Online publication date: Nov-2020
  • (2020)Adaptive level of autonomy for human-UAVs collaborative surveillance using situated fuzzy cognitive mapsChinese Journal of Aeronautics10.1016/j.cja.2020.03.03133:11(2835-2850)Online publication date: Nov-2020
  • (2019)Non-Technical Skills (NTS) Training for UAV OperatorsUnmanned Aerial Vehicles in Civilian Logistics and Supply Chain Management10.4018/978-1-5225-7900-7.ch001(1-32)Online publication date: 2019
  • (2019)A mixed integer programming (MIP) model for evaluating navigation and task planning of human---robot interactions (HRI)Intelligent Service Robotics10.1007/s11370-019-00275-w12:3(231-242)Online publication date: 1-Jul-2019
  • (2018)Common Metrics to Benchmark Human-Machine Teams (HMT): A ReviewIEEE Access10.1109/ACCESS.2018.28535606(38637-38655)Online publication date: 2018
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media