skip to main content
10.1145/3126686.3126690acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Real-Time Image-based Smoke Detection in Endoscopic Videos

Published: 23 October 2017 Publication History

Abstract

The nature of endoscopy as a type of minimally invasive surgery (MIS) requires surgeons to perform complex operations by merely inspecting a live camera feed. Inherently, a successful intervention depends upon ensuring proper working conditions, such as skillful camera handling, adequate lighting and removal of confounding factors, such as fluids or smoke. The latter is an undesirable byproduct of cauterizing tissue and not only constitutes a health hazard for the medical staff as well as the treated patients, it can also considerably obstruct the operating physician's field of view. Therefore, as a standard procedure the gaseous matter is evacuated by using specialized smoke suction systems that typically are activated manually whenever considered appropriate. We argue that image-based smoke detection can be employed to undertake such a decision, while as well being a useful indicator for relevant scenes in post-procedure analyses. This work represents a continued effort to previously conducted studies utilizing pre-trained convolutional neural networks (CNNs) and threshold-based saturation analysis. Specifically, we explore further methodologies for comparison and provide as well as evaluate a public dataset comprising over 100K smoke/non-smoke images extracted from the Cholec80 dataset, which is composed of 80 different cholecystectomy procedures. Having applied deep learning to merely 20K images of a custom dataset, we achieve Receiver Operating Characteristic (ROC) curves enclosing areas of over 0.98 for custom datasets and over 0.77 for the public dataset. Surprisingly, a fixed threshold for saturation-based histogram analysis still yields areas of over 0.78 and 0.75.

References

[1]
2006. Linux Mint 17.3 "Rosa" - Cinnamon (64-bit). (2006). https://linuxmint.com/edition.php?id=204 Accessed: 2017-03-28.
[2]
2017. OpenCV library. (2017). http://opencv.org/
[3]
2017. Python programming language. (2017). https://www.python.org/
[4]
O. S. Al Sahaf, I. Vega-Carrascal, F. O. Cunningham, J. P. McGrath, and F. J. Bloomfield. 2007. Chemical composition of smoke produced by high-frequency electrosurgery. Irish Journal of Medical Science 176, 3 (2007), 229--232.
[5]
Simone Calderara, Paolo Piccinini, and Rita Cucchiara. 2011. Vision based smoke detection system using image energy and color information. Machine Vision and Applications 22, 4 (jul 2011), 705--719.
[6]
Ming-Chang Liu Chen-Rui Chou. 2016. System and Method for Smoke Detection During Anatomical Surgery. (2016). https://www.google.com/patents/US20160239967
[7]
Seock Hwan Choi, Tae Gyun Kwon, Sung Kwang Chung, and Tae Hwan Kim. 2014. Surgical smoke may be a biohazard to surgeons performing laparoscopic surgery. Surgical Endoscopy and Other Interventional Techniques 28, 8 (2014), 2374--2380.
[8]
Yu Chunyu, Fang Jun, Wang Jinjun, and Zhang Yongming. 2010. Video Fire Smoke Detection Using Motion and Color Features. Fire Technology 46, 3 (jul 2010), 651--663.
[9]
Ioan Cosmescu. 1991. Automatic smoke evacuator system for a surgical laser apparatus and method therefor. (1991). https://www.google.com/patents/US5199944
[10]
Ioan Cosmescu. 2006. Automatic smoke evacuator and insufflation system for surgical procedures. (2006). https://www.google.com/patents/US20070249990
[11]
M. Dobrogowski, W. Wesołowski, M. Kucharska, A. Sapota, and L. Pomorski. 2014. Chemical composition of surgical smoke formed in the abdominal cavity during laparoscopic cholecystectomy - Assessment of the risk to the patient. International Journal of Occupational Medicine and Environmental Health 27, 2 (jan 2014), 314--325.
[12]
Ricardo J Ferrari, Hong Zhang, and C. Ronald Kube. 2007. Real-time detection of steam in video images. Pattern Recognition 40, 3 (2007), 1148--1159.
[13]
J. Gubbi, S. Marusic, and M. Palaniswami. 2009. Smoke detection in video using wavelets and support vector machines. Fire Safety Journal 44, 8 (2009), 1110--1115.
[14]
M. Häfner, A. Gangl, M. Liedlgruber, A. Uhl, A. Vécsei, and F. Wrba. 2009. Combining Gaussian Markov random fields with the discretewavelet transform for endoscopic image classification. In DSP 2009: 16th International Conference on Digital Signal Processing, Proceedings.
[15]
M. Hafner, A. Gangl, M. Liedlgruber, A. Uhl, A. Vecsei, and F. Wrba. 2010. Endoscopic Image Classification Using Edge-Based Features. In 2010 20th International Conference on Pattern Recognition. IEEE, 2724--2727.
[16]
M. Häfner, M. Liedlgruber, A. Uhl, A. Vécsei, and F. Wrba. 2012. Color treatment in endoscopic image classification using multi-scale local color vector patterns. Medical Image Analysis 16, 1 (2012), 75--86.
[17]
C. Hensman, D. Baty, R. G. Willis, and A. Cuschieri. 1998. Chemical composition of smoke produced by high-frequency electrosurgery in a closed gaseous environment. Surgical endoscopy (1998). http://www.springerlink.com/index/3PDVCC89D248BJT0.pdf
[18]
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 675--678.
[19]
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[20]
I. Kolesov, P. Karasev, A. Tannenbaum, and E. Haber. 2010. Fire and smoke detection in video with optimal mass transport based optical flow and neural networks. In 2010 IEEE International Conference on Image Processing. IEEE, 761-- 764.
[21]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
[22]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
[23]
S. Kudo, S. Hirota, T. Nakajima, S. Hosobe, H. Kusaka, T. Kobayashi, M. Himori, and A. Yagyuu. 1994. Colorectal tumours and pit pattern. Journal of clinical pathology 47, 10 (oct 1994), 880--5.
[24]
C. Y. Lee, C. T. Lin, C. T. Hong, and M. T. Su. 2012. SMOKE DETECTION USING SPATIAL AND TEMPORAL ANALYSES. International Journal of Innovative Computing Information and Control 8, 7A (2012), 4749--4770.
[25]
Andreas Leibetseder, Manfred Jürgen Primus, Stefan Petscharnig, and Klaus Schoeffmann. 2017. Image-based Smoke Detection in Laparoscopic Videos: Fourth International Workshop, CARE 2017, Held in Conjunction with MICCAI 2017, Quebec City, Canada, September 14, 2017, Accepted. Springer.
[26]
M. Liedlgruber and A. Uhl. 2009. Endoscopic image processing - an overview. In 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis. IEEE, 707--712.
[27]
Constantinos Loukas and Evangelos Georgiou. 2015. Smoke detection in endoscopic surgery videos: a first step towards retrieval of semantic events: Smoke detection in endoscopic surgery videos. The International Journal of Medical Robotics and Computer Assisted Surgery 11, 1 (mar 2015), 80--94.
[28]
Dietmar Mattes, Edah Silajdzic, Monika Mayer, Martin Horn, Daniel Scheidbach, Werner Wackernagel, Gerald Langmann, and Andreas Wedrich. 2010. Surgical smoke management for minimally invasive (micro)endoscopy: An experimental study. Surgical Endoscopy and Other Interventional Techniques 24, 10 (2010), 2492--2501.
[29]
T. Menes and H. Spivak. 2000. Laparoscopy: searching for the proper insufflation gas. Surgical endoscopy 14, 11 (nov 2000), 1050--6. http://www.ncbi.nlm.nih.gov/ /11116418
[30]
Bernd Münzer, Klaus Schoeffmann, and Laszlo Böszörmenyi. 2013. Detection of circular content area in endoscopic videos. In Computer-Based Medical Systems (CBMS), 2013 IEEE 26th International Symposium on. IEEE, 534--536.
[31]
Bernd Münzer, Klaus Schoeffmann, and Laszlo Böszörmenyi. 2017. Content-based processing and analysis of endoscopic images and videos: A survey. Multimedia Tools and Applications (2017).
[32]
J. A. Ojo and J. A. Oladosu. 2014. Video-based Smoke Detection Algorithms: A Chronological Survey. Computer Engineering and Intelligent Systems 5, 7 (2014), 38--50.
[33]
D. Ott. 1993. Smoke production and smoke reduction in endoscopic surgery: preliminary report. Endoscopic surgery and allied technologies 1, 4 (aug 1993), 230--2. http://www.ncbi.nlm.nih.gov/ /8050026
[34]
Sun Young Park and Dusty Sargent. 2016. Colonoscopic polyp detection using convolutional neural networks, Georgia D. Tourassi and Samuel G. Armato (Eds.). International Society for Optics and Photonics, 978528.
[35]
Stefan Petscharnig and Klaus Schöffmann. 2017. Deep Learning for Shot Classification in Gynecologic Surgery Videos. Vol. 10132. Springer International Publishing, Cham, 702--713. http://link.springer.com/10.1007/978-3-319-51811-4
[36]
Stefan Petscharnig and Klaus Schöffmann. 2017. Learning laparoscopic video shot classification for gynecological surgery. Multimedia Tools and Applications (apr 2017).
[37]
Klaus Schoeffmann, Marco A. Hudelist, and Jochen Huber. 2015. Video interaction tools: a survey of recent work. ACM Computing Surveys (CSUR) 48, 1 (2015), 14.
[38]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--9.
[39]
Hidekazu Takahashi, Makoto Yamasaki, Masashi Hirota, Yasuaki Miyazaki, Jeong Ho Moon, Yoshihito Souma, Masaki Mori, Yuichiro Doki, and Kiyokazu Nakajima. 2013. Automatic smoke evacuation in laparoscopic surgery: a simplified method for objective evaluation. Surgical Endoscopy 27, 8 (Aug. 2013), 2980--2987.
[40]
H. P. Thiébaud, M. G. Knize, P. A. Kuzmicky, D. P. Hsieh, and J. S. Felton. 1995. Airborne mutagens produced by frying beef, pork and a soy-based food. Food and Chemical Toxicology 33, 10 (1995), 821--828.
[41]
Hongda Tian, Wanqing Li, Lei Wang, and Philip Ogunbona. 2012. A novel video-based smoke detection method using image separation. In Proceedings - IEEE International Conference on Multimedia and Expo. 532--537.
[42]
B. Ugur Toreyin, Yigithan Dedeoglu, and A. Enis Cetin. 2006. Contour based smoke detection in video using wavelets. IEEE, 1--5.
[43]
Andru P. Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel de Mathelin, and Nicolas Padoy. 2017. EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE Transactions on Medical Imaging 36, 1 (jan 2017), 86--97.
[44]
Shiqian Wu, Feiniu Yuan, Yong Yang, Zhijun Fang, and Yuming Fang. 2015. Real-time image smoke detection using staircase searching-based dual threshold AdaBoost and dynamic analysis. IET Image Processing 9, 10 (oct 2015), 849--856.
[45]
Feiniu Yuan. 2011. Video-based smoke detection with histogram sequence of LBP and LBPV pyramids. Fire Safety Journal 46, 3 (2011), 132--139.

Cited By

View all
  • (2024)Transfer learning and Machine Learning Classification for Laparoscopic Video Distortion Detection2024 8th International Conference on Image and Signal Processing and their Applications (ISPA)10.1109/ISPA59904.2024.10536866(1-5)Online publication date: 21-Apr-2024
  • (2024)Smoke Classification in Laparoscopic Cholecystectomy Videos Incorporating Spatio-temporal InformationBildverarbeitung für die Medizin 202410.1007/978-3-658-44037-4_78(298-303)Online publication date: 20-Feb-2024
  • (2023)Multi-stages de-smoking model based on CycleGAN for surgical de-smokingInternational Journal of Machine Learning and Cybernetics10.1007/s13042-023-01875-w14:11(3965-3978)Online publication date: 5-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
Thematic Workshops '17: Proceedings of the on Thematic Workshops of ACM Multimedia 2017
October 2017
558 pages
ISBN:9781450354165
DOI:10.1145/3126686
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. cnn classification
  2. deep learning
  3. endoscopic surgery
  4. image processing
  5. smoke detection

Qualifiers

  • Research-article

Funding Sources

  • Carinthian Economic Promotion Fund
  • The European Regional Development Fund

Conference

MM '17
Sponsor:
MM '17: ACM Multimedia Conference
October 23 - 27, 2017
California, Mountain View, USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)32
  • Downloads (Last 6 weeks)3
Reflects downloads up to 10 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Transfer learning and Machine Learning Classification for Laparoscopic Video Distortion Detection2024 8th International Conference on Image and Signal Processing and their Applications (ISPA)10.1109/ISPA59904.2024.10536866(1-5)Online publication date: 21-Apr-2024
  • (2024)Smoke Classification in Laparoscopic Cholecystectomy Videos Incorporating Spatio-temporal InformationBildverarbeitung für die Medizin 202410.1007/978-3-658-44037-4_78(298-303)Online publication date: 20-Feb-2024
  • (2023)Multi-stages de-smoking model based on CycleGAN for surgical de-smokingInternational Journal of Machine Learning and Cybernetics10.1007/s13042-023-01875-w14:11(3965-3978)Online publication date: 5-Jun-2023
  • (2023)Real-time endoscopy haze removal: a synthetical methodMultimedia Tools and Applications10.1007/s11042-023-16375-w83:11(31195-31209)Online publication date: 18-Sep-2023
  • (2022)Endometriosis detection and localization in laparoscopic gynecologyMultimedia Tools and Applications10.1007/s11042-021-11730-181:5(6191-6215)Online publication date: 1-Feb-2022
  • (2021)Gamma Camera Imaging with Rotating Multi-Pinhole Collimator. A Monte Carlo Feasibility StudySensors10.3390/s2110336721:10(3367)Online publication date: 12-May-2021
  • (2021)Unsupervised Anomaly Detection for a Smart Autonomous Robotic Assistant Surgeon (SARAS) Using a Deep Residual AutoencoderIEEE Robotics and Automation Letters10.1109/LRA.2021.30972446:4(7256-7261)Online publication date: Oct-2021
  • (2021)Co-occurrence balanced time series classification for the semi-supervised recognition of surgical smokeInternational Journal of Computer Assisted Radiology and Surgery10.1007/s11548-021-02411-3Online publication date: 25-May-2021
  • (2021)Fire Detection by Parallel Classification of Fire and Smoke Using Convolutional Neural NetworkComputational Vision and Bio-Inspired Computing10.1007/978-981-33-6862-0_8(95-105)Online publication date: 15-Jun-2021
  • (2020)Improving endoscopic smoke detection with semi-supervised noisy student modelsCurrent Directions in Biomedical Engineering10.1515/cdbme-2020-00266:1Online publication date: 17-Sep-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media