skip to main content
10.1145/3078971.3079008acmconferencesArticle/Chapter ViewAbstractPublication PagesicmrConference Proceedingsconference-collections
research-article

Semi-Automatic Retrieval of Relevant Segments from Laparoscopic Surgery Videos

Published: 06 June 2017 Publication History

Abstract

Over the last decades, progress in medical technology and imaging technology enabled the technique of minimally invasive surgery. In addition, multimedia technologies allow for retrospective analyses of surgeries. The accumulated videos and images allow for a speed-up in documentation, easier medical case assessment across surgeons, training young surgeons, as well as they find the usage in medical research. Considering a surgery lasting for hours of routine work, surgeons only need to see short video segments of interest to assess a case. Surgeons do not have the time to manually extract video sequences of their surgeries from their big multimedia databases as they do not have the resources for this time-consuming task. The thesis deals with the questions of how to semantically classify video frames using Convolutional Neural Networks into different semantic concepts of surgical actions and anatomical structures. In order to achieve this goal, the capabilities of predefined CNN architectures and transfer learning in the laparoscopic video domain are investigated. The results are expected to improve by domain-specific adaptation of the CNN input layers, i.e. by fusion of the image with motion and relevance information. Finally, the thesis investigates to what extent surgeons' needs are covered with the proposed extraction of relevant scenes.

References

[1]
S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, and N. Navab. 2016. AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images. IEEE Transactions on Medical Imaging 35, 5 (2016), pages 1313--1321.
[2]
M. Anthimopoulos, S. Christodoulidis, L. Ebner, A. Christe, and S. Mougiakakou. 2016. Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network. IEEE Transactions on Medical Imaging 35, 5 (2016), pages 1207--1216.
[3]
Christian Beecks, Klaus Schoeffmann, Mathias Lux, Merih Seran Uysal, and Thomas Seidl. 2015. Endoscopic Video Retrieval: A Signature-based Approach for Linking Endoscopic Images with Video Segments. In Proceedings of the IEEE International Symposium on Multimedia 2015 (ISM 2015), Alberto Del Bimbo, Shu-Ching Chen, Haohong Wang, Heather Yu, and Roger Zimmermann (Eds.). IEEE, Los Alamitos, CA, pages 1--6.
[4]
Jennifer Roldan Carlos, Mathias Lux, Xavier Giro-i Nieto, Pia Munoz, and Nektarios Anagnostopoulos. 2015. Visual information retrieval in endoscopic video archives. In Content-Based Multimedia Indexing (CBMI), 2015 13th International Workshop on. IEEE, pages 1--6.
[5]
Qing Li, Weidong Cai, Xiaogang Wang, Yun Zhou, David Dagan Feng, and Mei Chen. 2014. Medical image classification with convolutional neural network. In Control Automation Robotics & Vision (ICARCV), 2014 13th International Conference on. IEEE, pages 844--848.
[6]
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen AWM van der Laak, Bram van Ginneken, and Clara I Sánchez. 2017. A Survey on Deep Learning in Medical Image Analysis. arXiv preprint arXiv:1702.05747 (2017).
[7]
Constantinos Loukas, Nikolaos Nikiteas, Dimitrios Schizas, and Evangelos Georgiou. 2016. Shot boundary detection in endoscopic surgery videos using a variational Bayesian framework. International Journal of Computer Assisted Radiology and Surgery 11, 11 (2016), pages 1937--1949. ISSN 1861--6429
[8]
Bernd Münzer, Klaus Schoeffmann, and Laszlo Böszörmenyi. 2017. Content-based processing and analysis of endoscopic images and videos: A survey. Multimedia Tools and Applications (2017), pages 1--40. ISSN 1573--7721
[9]
B. Münzer, K. Schoeffmann, and L. Böszörmenyi. 2013. Relevance Segmentation of Laparoscopic Videos. In 2013 IEEE International Symposium on Multimedia. pages 84--91.
[10]
B. Münzer, K. Schoeffmann, and L. Böszörmenyi. 2013. Relevance Segmentation of Laparoscopic Videos. In 2013 IEEE International Symposium on Multimedia. pages 84--91.
[11]
Sun Young Park and Dusty Sargent. 2016. Colonoscopic polyp detection using convolutional neural networks. In SPIE Medical Imaging. International Society for Optics and Photonics, pages 978528--978528.
[12]
M. J. Primus, K. Schoeffmann, and L. Böszörmenyi. 2015. Instrument classification in laparoscopic videos. In 2015 13th International Workshop on Content-Based Multimedia Indexing (CBMI). pages 1--6.
[13]
M. J. Primus, K. Schoeffmann, and nL. Böszörmenyi. 2016. Temporal segmentation of laparoscopic videos into surgical phases. In 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI). pages 1--6.
[14]
Yuchen Qiu, Yunzhi Wang, Shiju Yan, Maxine Tan, Samuel Cheng, Hong Liu, and Bin Zheng. 2016. An initial investigation on developing a new method to predict short-term breast cancer risk based on deep learning technology. In SPIE Medical Imaging. International Society for Optics and Photonics, pages 978521--978521.
[15]
G. Quellec, M. Lamard, B. Cochener, and G. Cazuguel. 2014. Real-Time Segmentation and Recognition of Surgical Tasks in Cataract Surgery Videos. IEEE Transactions on Medical Imaging 33, 12 (2014), pages 2352--2360.
[16]
Eduardo Ribeiro, Andreas Uhl, Georg Wimmer, and Michael Häfner. 2016. Transfer Learning for Colonic Polyp Classification Using Off-the-Shelf CNN Features. In International Workshop on Computer-Assisted and Robotic Endoscopy. Springer,bibinfopages1--13.
[17]
Ravi K Samala, Heang-Ping Chan, Lubomir M Hadjiiski, Kenny Cha, and Mark A Helvie. 2016. Deep-learning convolution neural network for computer-aided detection of microcalcifications in digital breast tomosynthesis. In SPIE Medical Imaging. International Society for Optics and Photonics, pages 97850Y--97850Y.
[18]
Klaus Schoeffmann, Christian Beecks, Mathias Lux, Merih Seran Uysal, and Thomas Seidl. 2016. Content-based retrieval in videos from laparoscopic surgery. Proc. SPIE 9786, pages 97861V--97861V--10.
[19]
H. C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, and R. M. Summers. 2016. Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning. IEEE Transactions on Medical Imaging 35, 5 (2016), pages 1285--1298.
[20]
N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, and J. Liang. 2016. Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Transactions on Medical Imaging 35, 5 (2016), pages 1299--1312.
[21]
A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. de Mathelin, and N. Padoy. 2017. EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE Transactions on Medical Imaging 36, 1 (2017), pages 86--97.
[22]
F. Xing, Y. Xie, and L. Yang. 2016. An Automatic Learning-Based Framework for Robust Nucleus Segmentation. IEEE Transactions on Medical Imaging 35, 2 (2016), pages 550--566.
[23]
Z. Yan, Y. Zhan, Z. Peng, S. Liao, Y. Shinagawa, S. Zhang, D. N. Metaxas, and X. S. Zhou. 2016. Multi-Instance Deep Learning: Discover Discriminative Local Anatomies for Bodypart Recognition. IEEE Transactions on Medical Imaging 35, 5 (2016), pages 1332--1343.

Cited By

View all
  • (2024)An intelligent surgical video retrieval for computer vision enhancement in medical diagnosis using deep learning techniquesMultimedia Tools and Applications10.1007/s11042-024-18813-9Online publication date: 29-May-2024
  • (2024)Automatic Retrieval of UAV Tilt Image and Image Attitude RecoveryFrontier Computing on Industrial Applications Volume 310.1007/978-981-99-9416-8_12(71-76)Online publication date: 30-Jan-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval
June 2017
524 pages
ISBN:9781450347013
DOI:10.1145/3078971
  • General Chairs:
  • Bogdan Ionescu,
  • Nicu Sebe,
  • Program Chairs:
  • Jiashi Feng,
  • Martha Larson,
  • Rainer Lienhart,
  • Cees Snoek
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 June 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. endoscopic image classification
  2. endoscopic video retrieval

Qualifiers

  • Research-article

Funding Sources

Conference

ICMR '17
Sponsor:

Acceptance Rates

ICMR '17 Paper Acceptance Rate 33 of 95 submissions, 35%;
Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)1
Reflects downloads up to 10 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)An intelligent surgical video retrieval for computer vision enhancement in medical diagnosis using deep learning techniquesMultimedia Tools and Applications10.1007/s11042-024-18813-9Online publication date: 29-May-2024
  • (2024)Automatic Retrieval of UAV Tilt Image and Image Attitude RecoveryFrontier Computing on Industrial Applications Volume 310.1007/978-981-99-9416-8_12(71-76)Online publication date: 30-Jan-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media