Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging

Abstract

In correlation-based time-of-flight (C-ToF) imaging systems, light sources with temporally varying intensities illuminate the scene. Due to global illumination, the temporally varying radiance received at the sensor is a combination of light received along multiple paths. Recovering scene properties (e.g., scene depths) from the received radiance requires separating these contributions, which is challenging due to the complexity of global illumination and the additional temporal dimension of the radiance.

We propose phasor imaging, a framework for performing fast inverse light transport analysis using C-ToF sensors. Phasor imaging is based on the idea that, by representing light transport quantities as phasors and light transport events as phasor transformations, light transport analysis can be simplified in the temporal frequency domain. We study the effect of temporal illumination frequencies on light transport and show that, for a broad range of scenes, global radiance (inter-reflections and volumetric scattering) vanishes for frequencies higher than a scene-dependent threshold. We use this observation for developing two novel scene recovery techniques. First, we present micro-ToF imaging, a ToF-based shape recovery technique that is robust to errors due to inter-reflections (multipath interference) and volumetric scattering. Second, we present a technique for separating the direct and global components of radiance. Both techniques require capturing as few as 3--4 images and minimal computations. We demonstrate the validity of the presented techniques via simulations and experiments performed with our hardware prototype.

References

1. X. Ai, R. Nock, J. G. Rarity, and N. Dahnoun. 2011. High-resolution random-modulation CW lidar. Appl. Optics 50, 22.
2. M. Akbulut, C. H. Chen, M. C. Hargis, A. M. Weiner, M. R. Mel-Loch, and J. M. Woodall. 2001. Digital communications above 1 Gb/s using 890-nm surface-emitting light-emitting diodes. IEEE Photon. Technol. Lett. 13, 1.
3. J. Busck and H. Heiselberg. 2004. High accuracy 3D laser radar. Proc. SPIE 5412.
4. B. Buttgen, M.-A. El Mechat, F. Lustenberger, and P. Seitz. 2007. Pseudonoise optical modulation for real-time 3-d imaging with minimum interference. IEEE Trans. Circ. Syst. 54, 10, 2109--2119.
5. B. Buxbaum, R. Schwarte, T. Ringbeck, M. Grothof, and X. Luan. 2002. Msm-pmd as correlation receiver in a new 3d-ranging system. Proc. SPIE 4546.
6. D. A. Carnegie, J. R. K. McClymont, A. P. P. Jongenelen, B. Drayto, A. A. Dorrington, and A. D. Payne. 2011. Design and construction of a configurable full-field range imaging system for mobile robotic applications. In New Developments and Applications in Sensing Technology. Springer, 133--155.
7. C. H. Chen, M. Hargis, J. M. Woodall, M. R. Melloch, J. S. Reynolds, E. Yablonovitch, and W. Wang. 1999. Ghz bandwidth gaas light-emitting diodes. Appl. Phys. Lett. 74, 21.
8. T. Chen, H. P. Seidel, and H. Lensch. 2008. Modulated phase-shifting for 3D scanning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08). 1--8.
9. V. Couture, N. Martin, and S. Roy. 2014. Unstructured light scanning robust to indirect illumination and depth discontinuities. Int. J. Comput. Vis. 108, 3, 204--221.
10. A. A. Dorrington, J. P. Godbaz, M. J. Cree, A. D. Payne, and L. V. Streeter. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. Proc. SPIE 7864.
11. R. Ferriere, J. Cussey, and J. Dudley. 2008. Time-of-flight range detection using low-frequency intensity modulation of a CW laser diode: Application to fiber length measurement. Optical Engin. 47, 9.
12. D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt. 2014. SRA: Fast removal of general multipath for ToF sensors. In Proceedings of the 13^th European Conference on Computer Vision (ECCV'14). 234--249.
13. S. Fuchs. 2010. Multipath interference compensation in time-of-flight camera images. In Proceedings of the 20^th International Conference on Pattern Recognition (ICPR'10). 3583--3586.
14. S. Fuchs, M. Suppa, and O. Hellwich. 2013. Compensation for multipath in Tof camera measurements supported by photometric calibration and environment integration. In Proceedings of the 9^th International Conference on Computer Vision Systems (ICVS'13). 31--41.
15. J. Godbaz, M. Cree, and A. Dorrington. 2008. Mixed pixel return separation for a full-field ranger. In Proceedings of the 23^rd International Conference on Image and Vision Computing New Zealand (IVCNZ'08). 1--6.
16. J. P. Godbaz, M. J. Cree, and A. A. Dorrington. 2009. Multiple return separation for a full-field ranger via continuous waveform modelling. Proc. SPIE 7251.
17. J. P. Godbaz, M. J. Cree, and A. A. Dorrington. 2012. Closed-form inverses for the mixed pixel/multipath interference problem in AMCW lidar. Proc. SPIE 8296.
18. S. B. Gokturk, H. Yalcin, and C. Bamji. 2004. A time-of-flight depth sensor - System description, issues and solutions. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04). 35.
19. J. Gu, T. Kobayashi, M. Gupta, and S. Nayar. 2011. Multiplexed illumination for scene recovery in the presence of global illumination. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'11). 691--698.
20. M. Gupta, A. Agrawal, A. Veeraraghavan, and S. G. Narasimhan. 2013a. A practical approach to 3d scanning in the presence of interreflections, subsurface scattering and defocus. Int. J. Comput. Vis. 102, 1--3, 33--55.
21. M. Gupta, Q. Yin, and S. K. Nayar. 2013b. Structured light in sunlight. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'13). 545--552.
22. M. Gupta and S. K. Nayar. 2012. Micro phase shifting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 813--820.
23. M. Gupta, Y. Tian, S. G. Narasimhan, and L. Zhang. 2009. (De)focusing on global light transport for active scene recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09). 2969--2976.
24. V. I. Gushov and Y. N. Solodkin. 1991. Automatic processing of fringe patterns in integer interferometers. Optics Lasers Engin. 14, 4--5.
25. S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'10). 553--560.
26. F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich. 2013. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. 32, 4.
27. J. Heinen, W. Huber, and W. Harth. 1976. Light-emitting diodes with a modulation bandwidth of more than 1 Ghz. Electron. Lett. 12, 21.
28. L. G. Henyey and J. L. Greenstein. 1941. Diffuse radiation in the galaxy. Astrophys. J. 93, 70--83.
29. D. Jimenez, D. Pizarro, M. Mazo, and S. Palazuelos. 2012. Modelling and correction of multipath interference in time of flight cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 893--900.
30. A. P. P. Jongenelen, D. G. Bailey, A. D. Payne, A. A. Dorrington, and D. A. Carnegie. 2011. Analysis of errors in tof range imaging with dual-frequency modulation. IEEE Trans. Instrument. Measure. 60, 5.
31. A. P. P. Jongenelen, D. Carnegie, A. D. Payne, and A. A. Dorrington. 2010. Maximizing precision over extended unambiguous range for Tof range imaging systems. In Proceedings of the IEEE Instrumentation and Measurement Technology Conference (I2MTC'10). 1575--1580.
32. A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. 2013. Coded time of flight cameras: Sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. 32, 6.
33. A. Kirmani, A. Benedetti, and P. Chou. 2013. SPUMIC: Simultaneous phase unwrapping and multipath interference cancellation in time-of-flight cameras using spectral methods. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME'13). 1--6.
34. A. Kirmani, T. Hutchison, J. Davis, and R. Raskar. 2009. Looking around the corner using transient imaging. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'09). 159--166.
35. W. Koechner. 1968. Optical ranging system employing a high power injection laser diode. IEEE Trans. Aerospace Electron. Syst. 4, 1.
36. R. Lange. 2000. 3d time-of-flight distance measurement with custom solid-state image sensors in cmos-ccd-technology. PhD thesis. http://d-nb.info/960293825/34.
37. R. Lange and P. Seitz. 2001. Solid state time-of-flight range camera. IEEE J. Quantum Electron. 37, 3.
38. N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. 30, 6.
39. S. G. Narasimhan, M. Gupta, C. Donner, R. Ramamoorthi, S. K. Nayar, and H. W. Jensen. 2006. Acquiring scattering properties of participating media by dilution. ACM Trans. Graph. 25, 3, 1003--1012.
40. S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. Graph. 25, 3, 935--944.
41. M. O'Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N.Kutulakos. 2014. Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Graph. 33, 4.
42. M. O'Toole, R. Raskar, and K. N. Kutulakos. 2012. Primal-dual coding to probe light transport. ACM Trans. Graph. 31, 4.
43. R. Pandharkar, A. Velten, A. Bardagjy, E. Lawson, M. Bawendi, and R. Raskar. 2011. Estimating motion and size of moving non-line-of-sight objects in cluttered environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11).265--272.
44. A. D. Payne, A. A. Dorrington, and M. J. Cree. 2010a. Illumination waveform optimization for time-of-flight range imaging cameras. Proc. SPIE 8085.
45. A. D. Payne, A. A. Dorrington, M. J. Cree, and D. A. Carnegie. 2010b. Improved measurement linearity and precision for AMCW time-of-flight range imaging cameras. Appl. Optics 49, 23.
46. D. Reddy, R. Ramamoorthi, and B. Curless. 2012. Frequency-space decomposition and acquisition of light transport under spatially varying illumination. In Proceedings of the 12^th European Conference on Computer Vision (ECCV'12). 596--610.
47. R. Schwarte. 2004. Breakthrough in multi-channel laser-radar technology providing thousands of high-sensitive lidar receivers on a chip. Proc. SPIE 5575.
48. R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte. 1997. New electro-optical mixing and correlating sensor: Facilities and applications of the photonic mixer device. Proc. SPIE 3100.
49. S. M. Seitz, Y. Matsushita, and K. N. Kutulakos. 2005. A theory of inverse light transport. In Proceedings of the 10^th IEEE International Conference on Computer Vision (ICCV'05). 1440--1447.
50. M. Takeda, Q. Gu, M. Kinoshita, H. Takai, and Y. Takahashi. 1997. Frequency-multiplex fourier-transform profilometry: A single-shot three-dimensional shape measurement of objects with large height discontinuities and/or surface isolations. Appl. Optics 36, 22.
51. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature 3, 745.
52. A. Velten, D. Wu, A. Jarabo, B. Masia, C. Barsi, C. Joshi, E. Lawson, M. Bawendi, D. Gutierrez, and R. Raskar. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 4.
53. G. Walter, C. Wu, H. Then, M. Feng, and N. Holonyak. 2009. Tilted-charge high speed (7 Ghz) light emitting diode. Appl. Phys. Lett. 94, 23.
54. C. H. Wu, G. Walter, H. Then, and M. Feng. 2010. Design and layout of multi Ghz operation of light emitting diodes. In Proceedings of the GaAs MAN-TECH Conference.
55. D. Wu, M. O'Toole, A. Velten, A. Agrawal, and R. Raskar. 2012a. Decomposing global light transport using time of flight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 366--373.
56. D. Wu, G. Wetzstein, C. Barsi, T. Willwacher, M. O'Toole, N. Naik, Q. Dai, K. Kutulakos, and R. Raskar. 2012b. Frequency analysis of transient light transport with applications in bare sensor imaging. In Proceedings of the 12^th European Conference on Computer Vision (ECCV'12).542--555.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Shape inference

  • Computer graphics

    • Shape modeling