skip to main content
research-article

Extracting depth and matte using a color-filtered aperture

Published: 01 December 2008 Publication History

Abstract

This paper presents a method for automatically extracting a scene depth map and the alpha matte of a foreground object by capturing a scene through RGB color filters placed in the camera lens aperture. By dividing the aperture into three regions through which only light in one of the RGB color bands can pass, we can acquir three shifted views of a scene in the RGB planes of an image in a single exposure. In other words, a captured image has depth-dependent color misalignment. We develop a color alignment measure to estimate disparities between the RGB planes for depth reconstruction. We also exploit color misalignment cues in our matting algorithm in order to disambiguate between the foreground and background regions even where their colors are similar. Based on the extracted depth and matte, the color misalignment in the captured image can be canceled, and various image editing operations can be applied to the reconstructed image, including novel view synthesis, postexposure refocusing, and composition over different backgrounds.

Supplementary Material

JPG File (a134-brando-mp4_hi.jpg)
MOV File (a134-brando-mp4_hi.mov)

References

[1]
Adelson, E. H., and Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. PAMI 14, 2, 99--106.
[2]
Amari, Y., and Adelson, E. H. 1992. Single-eye range estimation by using displaced apertures with color filters. In Proc. Int. Conf. Industrial Electronics, Control, Instrumentation, and Automation, vol. 3, 1588--1592.
[3]
Bando, Y., and Nishita, T. 2007. Towards digital refocusing from a single photograph. In Proc. Pacific Graphics, 363--372.
[4]
Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI 23, 11, 1222--1239.
[5]
Chang, I.-C., Huang, C.-L., Hsueh, W.-J., Lin, H.-C., Chen, C.-C., and Yeh, Y.-H. 2002. A novel 3-D hand-held camera based on tri-aperture lens. In Proc. SPIE 4925, 655--662.
[6]
Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proc. CVPR, vol. 2, 264--271.
[7]
Chuang, Y.-Y. 2004. New models and methods for matting and compositing. PhD thesis, University of Washington.
[8]
Georgeiv, T., Zheng, K. C., Curless, B., Salesin, D., Nayar, S., and Intwala, C. 2006. Spatio-angular resolution tradeoff in integral photography. In Proc. EGSR, 263--272.
[9]
Gradshteyn, I. S., and Ryzhik, I. M. 2000. Table of integrals, series, and products (sixth edition). Academic Press.
[10]
Grady, L. 2006. Random walks for image segmentation. IEEE Trans. PAMI 28, 11, 1768--1783.
[11]
Green, P., Sun, W., Matusik, W., and Durand, F. 2007. Multi-aperture photograpy. ACM Trans. Gr. 26, 3, 68:1--68:7.
[12]
Joshi, N., Matusik, W., and Avidan., S. 2006. Natural video matting using camera arrays. ACM Trans. Gr. 25, 3, 779--786.
[13]
Kim, J., Kolmogorov, V., and Zabih, R. 2003. Visual correspondence using energy minimization and mutual information. In Proc. ICCV, vol. 2, 1033--1040.
[14]
Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Gr. 26, 3, 70:1--70:9.
[15]
Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. PAMI 30, 2, 228--242.
[16]
Lewis, J. P. 1995. Fast template matching. In Proc. Vision Interface, 120--123.
[17]
Liang, C.-K., Lin, T.-H., Wong, B.-Y., Liu, C., and Chen, H. H. 2008. Programmable aperture photography: multiplexed light field acquisition. ACM Trans. Gr. 27, 3, 55:1--55:10.
[18]
McGuire, M., Matusik, M., Pfister, H., Durand, F., and Hughes, J. 2005. Defocus video matting. ACM Trans. Gr. 24, 3, 567--576.
[19]
McGuire, M., Matusik, W., and Yerazunis, W. 2006. Practical, real-time studio matting using dual imagers. In Proc. EGSR, 235--244.
[20]
Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P., 2005. Light field photography with a hand-held plenoptic camera. Tech. Rep. CSTR 2005-02, Stanford Computer Science, Apr.
[21]
Ng, R. 2005. Fourier slice photography. ACM Trans. Gr. 24, 3, 735--744.
[22]
Omer, I., and Werman, M. 2004. Color lines: image specific color representation. In Proc. CVPR, vol. 2, 946--953.
[23]
Raskar, R., Tumblin, J., Mohan, A., Agrawal, A., and Li, Y. 2006. Computational photography. In Proc. Eurographics STAR.
[24]
Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proc. ACM SIGGRAPH 96, 259--268.
[25]
Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Gr. 26, 3, 69:1--69:12.
[26]
Vlahos, P., 1971. Electronic composite photography. U. S. Patent 3,595,987.
[27]
Wang, J., and Cohen, M. F. 2007. Optimized color sampling for robust matting. In Proc. CVPR.
[28]
Wexler, Y., Fitzgibbon, A., and Zisserman, A. 2002. Bayesian estimation of layers from multiple images. In Proc. ECCV, 487--501.
[29]
Xiong, W., and Jia, J. 2007. Stereo matching on objects with fractional boundary. In Proc. CVPR.

Cited By

View all
  • (2024)Compact multispectral light field camera based on an inkjet-printed microlens array and color filter arrayOptics Express10.1364/OE.52164632:13(23510)Online publication date: 12-Jun-2024
  • (2023)Polarization Multi-Image Synthesis with Birefringent Metasurfaces2023 IEEE International Conference on Computational Photography (ICCP)10.1109/ICCP56744.2023.10233735(1-12)Online publication date: 28-Jul-2023
  • (2022)Deep fusion prior for plenoptic super-resolution all-in-focus imagingOptical Engineering10.1117/1.OE.61.12.12310361:12Online publication date: 1-Dec-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 27, Issue 5
December 2008
552 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/1409060
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2008
Published in TOG Volume 27, Issue 5

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. alpha matting
  2. color correlation
  3. color filters
  4. computational camera
  5. computational photography
  6. depth estimation

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)8
  • Downloads (Last 6 weeks)3
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Compact multispectral light field camera based on an inkjet-printed microlens array and color filter arrayOptics Express10.1364/OE.52164632:13(23510)Online publication date: 12-Jun-2024
  • (2023)Polarization Multi-Image Synthesis with Birefringent Metasurfaces2023 IEEE International Conference on Computational Photography (ICCP)10.1109/ICCP56744.2023.10233735(1-12)Online publication date: 28-Jul-2023
  • (2022)Deep fusion prior for plenoptic super-resolution all-in-focus imagingOptical Engineering10.1117/1.OE.61.12.12310361:12Online publication date: 1-Dec-2022
  • (2021)One-shot three-dimensional measurement method with the color mapping of light directionOSA Continuum10.1364/OSAC.4175114:3(840)Online publication date: 22-Feb-2021
  • (2021)REPAID: resolution-enhanced plenoptic all-in-focus imaging using deep neural networksOptics Letters10.1364/OL.43027246:12(2896)Online publication date: 14-Jun-2021
  • (2021)Can Optical Trojans Assist Adversarial Perturbations?2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)10.1109/ICCVW54120.2021.00019(122-131)Online publication date: Oct-2021
  • (2021)Accuracy in Depth Recovery and 3D Image Synthesis From Single Image Using Multi-Color Filter Aperture and Shallow Depth of FieldIEEE Access10.1109/ACCESS.2021.31098659(123528-123540)Online publication date: 2021
  • (2021)3D Image Generation From Single Image Using Color Filtered Aperture and 2.1D Sketch-A Computational 3D Imaging System and Qualitative AnalysisIEEE Access10.1109/ACCESS.2021.30899389(93580-93592)Online publication date: 2021
  • (2020)SIRA - An efficient method for retrieving stereo images from anaglyphsSignal Processing: Image Communication10.1016/j.image.2020.11586685(115866)Online publication date: Jul-2020
  • (2020)Traffic Image Dehazing Based on Wavelength Related Physical Imaging ModelSmart Multimedia10.1007/978-3-030-54407-2_34(403-413)Online publication date: 31-Jul-2020
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media