skip to main content
10.1145/3132390.3132395acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Surgical Action Retrieval for Assisting Video Review of Laparoscopic Skills

Published: 27 October 2017 Publication History

Abstract

An increasing number of surgeons promote video review of laparoscopic surgeries for detection of technical errors at an early stage as well as for training purposes. The reason behind is the fact that laparoscopic surgeries require specific psychomotor skills, which are difficult to learn and teach. The manual inspection of surgery video recordings is extremely cumbersome and time-consuming. Hence, there is a strong demand for automated video content analysis methods. In this work, we focus on retrieving surgical actions from video collections of gynecologic surgeries. We propose two novel dynamic content descriptors for similarity search and investigate a query-by-example approach to evaluate the descriptors on a manually annotated dataset consisting of 18 hours of video content. We compare several content descriptors including dynamic information of the segments as well as descriptors containing only spatial information of keyframes of the segments. The evaluation shows that our proposed dynamic content descriptors considering motion and spatial information from the segment achieve a better retrieval performance than static content descriptors ignoring temporal information of the segment at all. The proposed content descriptors in this work enable content-based video search for similar laparoscopic actions, which can be used to assist surgeons in evaluating laparoscopic surgical skills.

References

[1]
S. Atasoy, D. Mateus, J. Lallemand, A. Meining, G. Yang, and N. Navab. 2010. Endoscopic video manifolds. Medical Image Computing and Computer-Assisted Intervention - MICCAI 2010 (2010), 437--445.
[2]
A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky. 2014. Neural codes for image retrieval. In European Conference on Computer Vision. Springer, 584--599.
[3]
C. Beecks. 2013. Distance-based similarity models for content-based multimedia retrieval. Ph.D. Dissertation. RWTH Aachen University.
[4]
C. Beecks, S. Kirchh, and T. Seidl. 2013. Signature matching distance for content-based image retrieval. Proc. of the 3rd ACM conference on Int. conference on multimedia retrieval ICMR (2013), 41--48.
[5]
C. Beecks, J. Lokočc, T. Seidl, and T. Skopal. 2011. Indexing the signature quadratic form distance for efficient content-based multimedia retrieval. In Proc. of the 1st ACM Int. Conference on Multimedia Retrieval. ACM, 24.
[6]
C. Beecks, K. Schoeffmann, M. S. Lux, M. S. Uysal, and T. Seidl. 2015. Endoscopic video retrieval: A signature-based approach for linking endoscopic images with video segments. In 2015 IEEE Int. Symposium on Multimedia (ISM). IEEE, 33--38.
[7]
C. Beecks, M. S. Uysal, and T. Seidl. 2010. Signature Quadratic Form Distance. In Proc. of the 9th ACM Int. Conference on Image and Video Retrieval, CIVR. ACM, 438--445.
[8]
E. M. Bonrath, N. J. Dedy, L. E. Gordon, and T. P. Grantcharov. 2015. Comprehensive Surgical Coaching Enhances Surgical Skill in the Operating Room: A Randomized Controlled Trial. Annals of Surgery 262, 2 (2015).
[9]
E. M. Bonrath, B. Zevin, N. J. Dedy, and T. P. Grantcharov. 2013. Error rating tool to identify and analyse technical errors and events in laparoscopic surgery. British Journal of Surgery 100, 8 (2013), 1080--1088.
[10]
J. Bouguet. 2001. Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel Corporation 5, 1--10 (2001), 4.
[11]
M. K. Chmarra, N. H. Bakker, C. A. Grimbergen, and J. Dankelman. 2006. TrEndo, a device for tracking minimally invasive surgical instruments in training setups. Sensors and Actuators A: Physical 126, 2 (2006), 328--334.
[12]
R. Dahyot, F. Vilariño, and G. Lacey. 2008. Improving the quality of color colonoscopy videos. EURASIP Journal on Image and Video Processing 2008, 1 (2008), 1--7.
[13]
D. DeMenthon and D. Doermann. 2003. Video Retrieval Using Spatio-temporal Descriptors. In Proc. of the Eleventh ACM Int. Conference on Multimedia (MULTIMEDIA '03). ACM, 508--517.
[14]
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. 2014. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In ICML. 647--655.
[15]
Z. Droueche, M. Lamard, G. Cazuguel, G. Quellec, C. Roux, and B. Cochener. 2012. Motion-based video retrieval with application to computer-assisted retinal surgery. In 2012 Annual Int. Conference of the IEEE Engineering in Medicine and Biology Society. 4962--4965.
[16]
I. C. Duta, J. R. Uijlings, T. A. Nguyen, K. Aizawa, A. G. Hauptmann, B. Ionescu, and N. Sebe. 2016. Histograms of Motion Gradients for Real-time Video Classification. In 2016 14th Int. Workshop on Content-Based Multimedia Indexing (CBMI). IEEE, 1--6.
[17]
G. E. Herrera-Almario, K. Kirk, V. T. Guerrero, K. Jeong, S. Kim, and G. G. Hamad. 2016. The effect of video review of resident laparoscopic surgical skills measured by self- and external assessment. The American Journal of Surgery 211 (2016), 315--320.
[18]
T. Horeman, J. Dankelman, F. W. Jansen, and J. J. van den Dobbelsteen. 2014. Assessment of Laparoscopic Skills Based on Force and Motion Parameters. IEEE Transactions on Biomedical Engineering 61, 3 (2014), 805--813.
[19]
M. A. Hudelist, H. Husslein, B. Munzer, S. Kletz, and K. Schoeffmann. 2017. A Tool to Support Surgical Quality Assessment. In 2017 IEEE Third Int. Conference on Multimedia Big Data (BigMM). IEEE, 238--239.
[20]
H. Husslein, L. Shirreff, E. M. Shore, G. G. Lefebvre, and T. P. Grantcharov. 2015. The Generic Error Rating Tool: A Novel Approach to Assessment of Performance and Surgical Education in Gynecologic Laparoscopy. Journal of Surgical Education (2015).
[21]
B. Ionescu, C. Vertan, and L. Florea. 2011. Automatic Abstraction of Laparoscopic Medical Footage Through Visual Activity Analysis. In E-Health and Bioengineering Conference. {IEEE}, 1--4.
[22]
G. Islam, K. Kahol, B. Li, M. Smith, and V. L. Patel. 2016. Affordable, web-based surgical skill training and evaluation tool. Journal of Biomedical Informatics 59 (2016), 102--114.
[23]
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proc. of the 22Nd ACM Int. Conference on Multimedia (MM '14). ACM, 675--678.
[24]
W. E. Jr. Kelley. 2008. The evolution of laparoscopy and the revolution in surgery in the decade of the 1990s. JSLS : Journal of the Society of Laparoendoscopic Surgeons 12, 4 (2008), 351--7.
[25]
A. Krizhevsky, I. Sutskever, and G. E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25. Curran Associates, Inc., 1097--1105.
[26]
I. Laptev. 2005. On Space-Time Interest Points. Int. Journal of Computer Vision 64, 2 (2005), 107--123.
[27]
H. C. Lin, I. Shafran, D. Yuh, and G. D. Hager. 2006. Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions. Computer Aided Surgery 11, 5 (2006), 220--230.
[28]
S. J. McKenna, H. Nait Charif, and T. Frank. 2005. Towards video understanding of laparoscopic surgery: Instrument tracking. Proc. of Image and Vision Computing (2005).
[29]
B. Münzer, K. Schoeffmann, and L. Böszörmenyi. 2013. Relevance Segmentation of Laparoscopic Videos. In 2013 IEEE Int. Symposium on Multimedia. 84--91.
[30]
B. Münzer, K. Schoeffmann, and L. Böszörmenyi. 2017. Content-based processing and analysis of endoscopic images and videos: A survey. Multimedia Tools and Applications (2017), 1--40.
[31]
J. Oh, S. Hwang, J. Lee, W. Tavanapong, J. Wong, and P. C. De Groen. 2007. Informative frame classification for endoscopy video. Medical Image Analysis 11, 2 (2007), 110--127.
[32]
I. Oropesa, P. Sánchez-González, M. K. Chmarra, P. Lamata, A. Fernández, J. A. Sánchez-Margallo, F. W. Jansen, J. Dankelman, F. M. Sánchez-Margallo, and E. J. Gómez. 2013. EVA: Laparoscopic Instrument Tracking Based on Endoscopic Video Analysis for Psychomotor Skills Assessment. Surgical Endoscopy 27, 3 (2013), 1029--1039.
[33]
S. Petscharnig and K. Schoeffmann. 2017. Deep Learning for Shot Classification in Gynecologic Surgery Videos. In MultiMedia Modeling: 23rd In. Conference, MMM 2017, Reykjavik, Iceland, January 4-6, 2017, Proc., Part I. Springer Int. Publishing, 702--713.
[34]
S. Petscharnig and K Schoeffmann. 2017. Learning laparoscopic video shot classification for gynecological surgery. Multimedia Tools and Applications (2017).
[35]
M. J. Primus, K. Schoeffmann, and L. Böszörmenyi. 2015. Instrument Classification in Laparoscopic Videos. In Content-Based Multimedia Indexing (CBMI), 2015 13th Int. Workshop on. 1--6.
[36]
C. E. Reiley, H. C. Lin, D. D. Yuh, and G. D. Hager. 2011. Review of methods for objective surgical skill evaluation. Surgical Endoscopy 25, 2 (2011), 356--366.
[37]
J. Rosen, B. Hannaford, C. G. Richards, and M. N. Sinanan. 2001. Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. Biomedical Engineering, IEEE Transactions on 48, 5 (2001), 579--591.
[38]
Y. Rubner, C. Tomasi, and L. J. Guibas. 2000. The Earth Mover's Distance as a Metric for Image Retrieval. Int. Journal of Computer Vision 40, 2 (2000), 99--121.
[39]
C. Saint-Pierre, J. Boisvert, G. Grimard, and F. Cheriet. 2011. Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images. Machine Vision and Applications 22, 1 (2011), 171--180.
[40]
K. Schoeffmann, C. Beecks, M. Lux, M. S. Uysal, and T. Seidl. 2016. Content-based retrieval in videos from laparoscopic surgery. In SPIE Medical Imaging. Int. Society for Optics and Photonics, 97861V--97861V.
[41]
K. Schoeffmann, M. Del Fabro, T. Szkaliczki, L. Böszörmenyi, and J. Keckstein. 2014. Keyframe extraction in endoscopic video. Multimedia Tools and Applications (2014), 1--20.
[42]
K. Schoeffmann, M. Lux, M. Taschwer, and L. Böszörmenyi. 2009. Visualization of video motion in context of video browsing. In 2009 IEEE Int. Conference on Multimedia and Expo. 658--661.
[43]
K. Schulmann, S. Hollerbach, K. Kraus, J. Willert, T. Vogel, G. Moslein, C. Pox, M. Reiser, A. Reinacher-Schick, and W. Schmiegel. 2005. Feasibility and Diagnostic Utility of Video Capsule Endoscopy for the Detection of Small Bowel Polyps in Patients with Hereditary Polyposis Syndromes. Am. J. Gastroenterol 100, 1 (2005), 27--37.
[44]
R. M. Summers, C. D. Johnson, L. M. Pusanik, J. D. Malley, A. M. Youssef, and J. E. Reed. 2001. Automated Polyp Detection at CT Colonography: Feasibility Assessment in a Human Population 1. Radiology 219, 1 (2001), 51--59.
[45]
A. P. Twinanda, M. De Mathelin, and N. Padoy. 2014. Fisher kernel based task boundary retrieval in laparoscopic database with single video query. In Int. Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 409--416.
[46]
A. P. Twinanda, J. Marescaux, M. de Mathelin, and N. Padoy. 2015. Classification approach for automatic laparoscopic video database organization. Int. journal of computer assisted radiology and surgery 10, 9 (2015), 1449--1460.
[47]
Y. Wang, W. Tavanapong, J. Wong, J. Oh, and P. C. De Groen. 2013. Near real-time retroflexion detection in colonoscopy. IEEE Journal of biomedical and health informatics 17, 1 (2013), 143--152.

Cited By

View all
  • (2022)Hybrid Spatiotemporal Contrastive Representation Learning for Content-Based Surgical Video RetrievalElectronics10.3390/electronics1109135311:9(1353)Online publication date: 24-Apr-2022
  • (2021)Content based Surgical Video Retrieval via Multi-Deep Features Fusion2021 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT)10.1109/CONECCT52877.2021.9622562(1-5)Online publication date: 9-Jul-2021
  • (2018)On Reducing Effort in Evaluating Laparoscopic SkillsProceedings of the 26th ACM international conference on Multimedia10.1145/3240508.3243934(815-819)Online publication date: 15-Oct-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MultiEdTech '17: Proceedings of the 2017 ACM Workshop on Multimedia-based Educational and Knowledge Technologies for Personalized and Social Online Training
October 2017
38 pages
ISBN:9781450355087
DOI:10.1145/3132390
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. feature signatures
  2. laparoscopic video
  3. medical endoscopy
  4. motion analysis
  5. similarity search
  6. video retrieval

Qualifiers

  • Research-article

Funding Sources

  • Carinthian Economic Promotion Fund
  • European Regional Development Fund

Conference

MM '17
Sponsor:
MM '17: ACM Multimedia Conference
October 27, 2017
California, Mountain View, USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 10 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Hybrid Spatiotemporal Contrastive Representation Learning for Content-Based Surgical Video RetrievalElectronics10.3390/electronics1109135311:9(1353)Online publication date: 24-Apr-2022
  • (2021)Content based Surgical Video Retrieval via Multi-Deep Features Fusion2021 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT)10.1109/CONECCT52877.2021.9622562(1-5)Online publication date: 9-Jul-2021
  • (2018)On Reducing Effort in Evaluating Laparoscopic SkillsProceedings of the 26th ACM international conference on Multimedia10.1145/3240508.3243934(815-819)Online publication date: 15-Oct-2018
  • (2018)Evaluation of Visual Content Descriptors for Supporting Ad-Hoc Video Search Tasks at the Video Browser ShowdownMultiMedia Modeling10.1007/978-3-319-73603-7_17(203-215)Online publication date: 13-Jan-2018

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media