skip to main content
10.1145/1457515.1409087acmconferencesArticle/Chapter ViewAbstractPublication Pagessiggraph-asiaConference Proceedingsconference-collections
research-article

Extracting depth and matte using a color-filtered aperture

Published: 01 December 2008 Publication History

Abstract

This paper presents a method for automatically extracting a scene depth map and the alpha matte of a foreground object by capturing a scene through RGB color filters placed in the camera lens aperture. By dividing the aperture into three regions through which only light in one of the RGB color bands can pass, we can acquir three shifted views of a scene in the RGB planes of an image in a single exposure. In other words, a captured image has depth-dependent color misalignment. We develop a color alignment measure to estimate disparities between the RGB planes for depth reconstruction. We also exploit color misalignment cues in our matting algorithm in order to disambiguate between the foreground and background regions even where their colors are similar. Based on the extracted depth and matte, the color misalignment in the captured image can be canceled, and various image editing operations can be applied to the reconstructed image, including novel view synthesis, postexposure refocusing, and composition over different backgrounds.

Supplementary Material

MOV File (a134-brando-mp4_hi.mov)

References

[1]
Adelson, E. H., and Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. PAMI 14, 2, 99--106.
[2]
Amari, Y., and Adelson, E. H. 1992. Single-eye range estimation by using displaced apertures with color filters. In Proc. Int. Conf. Industrial Electronics, Control, Instrumentation, and Automation, vol. 3, 1588--1592.
[3]
Bando, Y., and Nishita, T. 2007. Towards digital refocusing from a single photograph. In Proc. Pacific Graphics, 363--372.
[4]
Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI 23, 11, 1222--1239.
[5]
Chang, I.-C., Huang, C.-L., Hsueh, W.-J., Lin, H.-C., Chen, C.-C., and Yeh, Y.-H. 2002. A novel 3-D hand-held camera based on tri-aperture lens. In Proc. SPIE 4925, 655--662.
[6]
Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proc. CVPR, vol. 2, 264--271.
[7]
Chuang, Y.-Y. 2004. New models and methods for matting and compositing. PhD thesis, University of Washington.
[8]
Georgeiv, T., Zheng, K. C., Curless, B., Salesin, D., Nayar, S., and Intwala, C. 2006. Spatio-angular resolution tradeoff in integral photography. In Proc. EGSR, 263--272.
[9]
Gradshteyn, I. S., and Ryzhik, I. M. 2000. Table of integrals, series, and products (sixth edition). Academic Press.
[10]
Grady, L. 2006. Random walks for image segmentation. IEEE Trans. PAMI 28, 11, 1768--1783.
[11]
Green, P., Sun, W., Matusik, W., and Durand, F. 2007. Multi-aperture photograpy. ACM Trans. Gr. 26, 3, 68:1--68:7.
[12]
Joshi, N., Matusik, W., and Avidan., S. 2006. Natural video matting using camera arrays. ACM Trans. Gr. 25, 3, 779--786.
[13]
Kim, J., Kolmogorov, V., and Zabih, R. 2003. Visual correspondence using energy minimization and mutual information. In Proc. ICCV, vol. 2, 1033--1040.
[14]
Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Gr. 26, 3, 70:1--70:9.
[15]
Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. PAMI 30, 2, 228--242.
[16]
Lewis, J. P. 1995. Fast template matching. In Proc. Vision Interface, 120--123.
[17]
Liang, C.-K., Lin, T.-H., Wong, B.-Y., Liu, C., and Chen, H. H. 2008. Programmable aperture photography: multiplexed light field acquisition. ACM Trans. Gr. 27, 3, 55:1--55:10.
[18]
McGuire, M., Matusik, M., Pfister, H., Durand, F., and Hughes, J. 2005. Defocus video matting. ACM Trans. Gr. 24, 3, 567--576.
[19]
McGuire, M., Matusik, W., and Yerazunis, W. 2006. Practical, real-time studio matting using dual imagers. In Proc. EGSR, 235--244.
[20]
Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P., 2005. Light field photography with a hand-held plenoptic camera. Tech. Rep. CSTR 2005-02, Stanford Computer Science, Apr.
[21]
Ng, R. 2005. Fourier slice photography. ACM Trans. Gr. 24, 3, 735--744.
[22]
Omer, I., and Werman, M. 2004. Color lines: image specific color representation. In Proc. CVPR, vol. 2, 946--953.
[23]
Raskar, R., Tumblin, J., Mohan, A., Agrawal, A., and Li, Y. 2006. Computational photography. In Proc. Eurographics STAR.
[24]
Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proc. ACM SIGGRAPH 96, 259--268.
[25]
Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Gr. 26, 3, 69:1--69:12.
[26]
Vlahos, P., 1971. Electronic composite photography. U. S. Patent 3,595,987.
[27]
Wang, J., and Cohen, M. F. 2007. Optimized color sampling for robust matting. In Proc. CVPR.
[28]
Wexler, Y., Fitzgibbon, A., and Zisserman, A. 2002. Bayesian estimation of layers from multiple images. In Proc. ECCV, 487--501.
[29]
Xiong, W., and Jia, J. 2007. Stereo matching on objects with fractional boundary. In Proc. CVPR.

Cited By

View all
  • (2024)Depth Estimation From a Single Optical Encoded Image Using a Learned Colored-Coded ApertureIEEE Transactions on Computational Imaging10.1109/TCI.2024.339670010(752-761)Online publication date: 2024
  • (2021)Learning to Estimate Kernel Scale and Orientation of Defocus Blur with Asymmetric Coded ApertureICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP39728.2021.9413920(1465-1469)Online publication date: 6-Jun-2021
  • (2020)Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR42600.2020.00254(2462-2471)Online publication date: Jun-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papers
December 2008
581 pages
ISBN:9781450318310
DOI:10.1145/1457515
  • Editor:
  • John C. Hart
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. alpha matting
  2. color correlation
  3. color filters
  4. computational camera
  5. computational photography
  6. depth estimation

Qualifiers

  • Research-article

Conference

SIGGRAPH '08
Sponsor:

Acceptance Rates

SIGGRAPH Asia '08 Paper Acceptance Rate 59 of 320 submissions, 18%;
Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Depth Estimation From a Single Optical Encoded Image Using a Learned Colored-Coded ApertureIEEE Transactions on Computational Imaging10.1109/TCI.2024.339670010(752-761)Online publication date: 2024
  • (2021)Learning to Estimate Kernel Scale and Orientation of Defocus Blur with Asymmetric Coded ApertureICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP39728.2021.9413920(1465-1469)Online publication date: 6-Jun-2021
  • (2020)Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR42600.2020.00254(2462-2471)Online publication date: Jun-2020
  • (2020)Punching holes in light: recent progress in single-shot coded-aperture optical imagingReports on Progress in Physics10.1088/1361-6633/abaf4383:11(116101)Online publication date: 27-Oct-2020
  • (2017)BibliographyFrom Photon to Pixel10.1002/9781119402442.biblio(417-437)Online publication date: 11-Feb-2017
  • (2016)Video Depth-from-Defocus2016 Fourth International Conference on 3D Vision (3DV)10.1109/3DV.2016.46(370-379)Online publication date: Oct-2016
  • (2015)BibliographyFrom Photon to Pixel10.1002/9781119238447.biblio(413-433)Online publication date: 30-Oct-2015
  • (2013)Anaglyph decomposition using disparity mapACM SIGGRAPH 2013 Posters10.1145/2503385.2503452(1-1)Online publication date: 21-Jul-2013
  • (2013)Design of a Chromatic 3D Camera with an End-to-End Performance Model ApproachProceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops10.1109/CVPRW.2013.140(953-960)Online publication date: 23-Jun-2013
  • (2012)Optimized anaglyph colorizationSIGGRAPH Asia 2012 Technical Briefs10.1145/2407746.2407755(1-4)Online publication date: 28-Nov-2012
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media