skip to main content
10.1145/3123266.3123360acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Semi-Dense Depth Interpolation using Deep Convolutional Neural Networks

Authors Info & Claims
Published:19 October 2017Publication History

ABSTRACT

With advances of recent technologies, augmented reality systems and autonomous vehicles gained a lot of interest from academics and industry. Both these areas rely on scene geometry understanding, which usually requires depth map estimation. However, in case of systems with limited computational resources, such as smartphones or autonomous robots, high resolution dense depth map estimation may be challenging. In this paper, we study the problem of semi-dense depth map interpolation along with low resolution depth map upsampling. We present an end-to-end learnable residual convolutional neural network architecture that achieves fast interpolation of semi-dense depth maps with different sparse depth distributions: uniform, sparse grid and along intensity image gradient. We also propose a loss function combining classical mean squared error with perceptual loss widely used in intensity image super-resolution and style transfer tasks. We show that with some modifications, this architecture can be used for depth map super-resolution. Finally, we evaluate our results on both synthetic and real data, and consider applications for autonomous vehicles and creating AR/MR video games.

References

  1. Oisin Mac Aodha, Neill D. F. Campbell, Arun Nair, and Gabriel J. Brostow. 2012. Patch Based Synthesis for Single Depth Image Super-resolution Proceedings of the 12th European Conference on Computer Vision - Volume Part III (ECCV'12). Springer-Verlag, Berlin, Heidelberg, 71--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Sebastian Brandt, Elem Güzel Kalayci, Roman Kontchakov, Vladislav Ryzhikov, Guohui Xiao, and Michael Zakharyaschev. 2017. Ontology-Based Data Access with a Horn Fragment of Metric Temporal Logic Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4--9, 2017, San Francisco, California, USA., bibfieldeditorSatinder P. Singh and Shaul Markovitch (Eds.). AAAI Press, NY, 1070--1076. http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14881Google ScholarGoogle Scholar
  3. Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black. 2012. A Naturalistic Open Source Movie for Optical Flow Evaluation Proceedings of the 12th European Conference on Computer Vision - Volume Part VI (ECCV'12). Springer-Verlag, Berlin, Heidelberg, 611--625. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 Vol. 1511.07289 (2015), 1--14.Google ScholarGoogle Scholar
  5. James Diebel and Sebastian Thrun. 2005. An Application of Markov Random Fields to Range Sensing Proceedings of the 18th International Conference on Neural Information Processing Systems (NIPS'05). MIT Press, Cambridge, MA, USA, 291--298. http://dl.acm.org/citation.cfm?id=2976248.2976285 Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a Deep Convolutional Network for Image Super-Resolution. Springer International Publishing, Cham, 184--199.Google ScholarGoogle Scholar
  7. Jakob Engel, Vladlen Koltun, and Daniel Cremers. 2016. Direct sparse odometry. arXiv preprint arXiv:1607.02565 Vol. 1607.02565 (2016), 1--17.Google ScholarGoogle Scholar
  8. Jakob Engel, Thomas Schöps, and Daniel Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. Springer International Publishing, Cham, 834--849.Google ScholarGoogle Scholar
  9. D. Ferstl, C. Reinbacher, R. Ranftl, M. Ruether, and H. Bischof. 2013. Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation 2013 IEEE International Conference on Computer Vision. IEEE, New York, NY, USA, 993--1000. /dl.acm.org/citation.cfm?id=938978.939190 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 (ICML'16). JMLR.org, New York, NY, USA, 1349--1357. http://dl.acm.org/citation.cfm?id=3045390.3045533 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. L. Wang, H. Wu, and C. Pan. 2014. Fast Image Upsampling via the Displacement Field. IEEE Transactions on Image Processing Vol. 23, 12 (Dec. 2014), 5123--5135. showISSN1057-7149Google ScholarGoogle ScholarCross RefCross Ref
  12. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing Vol. 13, 4 (2004), 600--612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma. 2010. Image Super-resolution via Sparse Representation. Trans. Img. Proc., Vol. 19, 11 (Nov.. 2010), 2861--2873. showISSN1057--7149 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Raymond Yeh, Chen Chen, Teck Yian Lim, Mark Hasegawa-Johnson, and Minh N Do. 2016. Semantic Image Inpainting with Perceptual and Contextual Losses. arXiv preprint arXiv:1607.07539 Vol. 1607.07539 (2016), 1--10.Google ScholarGoogle Scholar

Index Terms

  1. Semi-Dense Depth Interpolation using Deep Convolutional Neural Networks

                        Recommendations

                        Comments

                        Login options

                        Check if you have access through your login credentials or your institution to get full access on this article.

                        Sign in
                        • Published in

                          cover image ACM Conferences
                          MM '17: Proceedings of the 25th ACM international conference on Multimedia
                          October 2017
                          2028 pages
                          ISBN:9781450349062
                          DOI:10.1145/3123266

                          Copyright © 2017 ACM

                          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

                          Publisher

                          Association for Computing Machinery

                          New York, NY, United States

                          Publication History

                          • Published: 19 October 2017

                          Permissions

                          Request permissions about this article.

                          Request Permissions

                          Check for updates

                          Qualifiers

                          • research-article

                          Acceptance Rates

                          MM '17 Paper Acceptance Rate189of684submissions,28%Overall Acceptance Rate995of4,171submissions,24%

                          Upcoming Conference

                          MM '24
                          MM '24: The 32nd ACM International Conference on Multimedia
                          October 28 - November 1, 2024
                          Melbourne , VIC , Australia

                        PDF Format

                        View or Download as a PDF file.

                        PDF

                        eReader

                        View online with eReader.

                        eReader