Light field microscopy

ABSTRACT

By inserting a microlens array into the optical train of a conventional microscope, one can capture light fields of biological specimens in a single photograph. Although diffraction places a limit on the product of spatial and angular resolution in these light fields, we can nevertheless produce useful perspective views and focal stacks from them. Since microscopes are inherently orthographic devices, perspective views represent a new way to look at microscopic specimens. The ability to create focal stacks from a single photograph allows moving or light-sensitive specimens to be recorded. Applying 3D deconvolution to these focal stacks, we can produce a set of cross sections, which can be visualized using volume rendering. In this paper, we demonstrate a prototype light field microscope (LFM), analyze its optical performance, and show perspective views, focal stacks, and reconstructed volumes for a variety of biological specimens. We also show that synthetic focusing followed by 3D deconvolution is equivalent to applying limited-angle tomography directly to the 4D light field.

REFERENCES

• 1: Edward H. Adelson , John Y. A. Wang, Single Lens Stereo with a Plenoptic Camera, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.14 n.2, p.99-106, February 1992 [doi>10.1109/34.121783 ]
• 2: Agard, D. A. 1984. Optical sectioning microscopy: Cellular architecture in three dimensions. Ann. Rev. Biophys. Bioeng 13, 191--219.
• 3: Andersen, A. H., Kak, A. C., 1984. Simultaneous algebraic reconstruction technique (SART): A superior implementation of the ART algorithm. Ultrasonic Imaging 6, 81--94.
• 4: Arridge, S. R. 2001. Methods for the inverse problem in optical tomography. Proc. Waves and Imaging Through Complex Media. Kluwer, 307--329.
• 5: Kenneth R. Castleman, Digital Image Processing, Prentice Hall Professional Technical Reference, 1979
• 6: Chamgoulov, R. O., Lane, P. M., Macaulay, C. E. 2004. Optical computed-tomography microscope using digital spatial light modulation. Proc. SPIE 5324, 182--190.
• 7: Colsher, J. G. 1980. Fully three-dimensional positron emission tomography. Phys. Med. Biol. 25, 1, 103--115.
• 8: Corle, T. R., Kino, G. S. 1996. Confocal Scanning Optical Microscopy and Related Imaging Systems. Academic Press.
• 9: Ellis, G. W. 1966. Holomicrography: transformation of image during reconstruction a posteriori. Science 143, 1195--1196.
• 10: Goldberg, N. 1992. Camera technology: the dark side of the lens. Academic Press.
• 11: Goodman, J. 1996. Introduction to Fourier optics. 2nd edition, McGraw-Hill.
• 12: Gustafsson, M. G. L. 2005. Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution. Proc. National Academy of Sciences 102, 37.
• 13: Holmes, T. J., Bhattacharyya, S., et al. 1995. Light microscopic images reconstructed by maximum likelihood deconvolution. In Handbook of Biological Confocal Microscopy, ed. J. B. Pawley, Plenum Press, 389--402.
• 14: Inoue, S., Oldenbourg, R. 1995. Microscopes. In Handbook of Optics, 2nd edition, McGraw-Hill.
• 15: Inoue, S. and Spring, K. R. 1997. Video Microscopy. 2nd edition, Plenum Press.
• 16: Gabor, D. 1948. A new microscopic principle. Nature 161, 777--778.
• 17: Aaron Isaksen , Leonard McMillan , Steven J. Gortler, Dynamically reparameterized light fields, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, p.297-306, July 2000 [doi>10.1145/344779.344929]
• 18: Bahram Javidi , Fumio Okano, Three-Dimensional Television, Video and Display Technology, Springer-Verlag New York, Inc., Secaucus, NJ, 2002
• 19: Kak, A. C., Slaney, M. 1988. Principles of Computerized Tomographic Imaging. IEEE Press.
• 20: Kawata, S., Nakamura, 0., Minami, S. 1987. Optical microscope tomography. I. Support constraint. J. Opt. Soc. Am. A 4, 1, 292--297.
• 21: Kingslake, R. 1983. Optical system design. Academic Press.
• 22: Marc Levoy , Pat Hanrahan, Light field rendering, Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, p.31-42, August 1996 [doi>10.1145/237170.237199]
• 23: Marc Levoy , Billy Chen , Vaibhav Vaish , Mark Horowitz , Ian McDowall , Mark Bolas, Synthetic aperture confocal imaging, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 24: Markham, J., Conchello, J.-A. 2001. Artefacts in restored images due to intensity loss in three-dimensional fluorescence microscopy. J. Microscopy 204, 2, 93--98.
• 25: Mcnally, J. G., Preza, C., Conchello, J. A., Thomas, L. J. Jr. 1994. Artifacts in computational optical-sectioning microscopy. J. Opt. Soc. Am. A 11, 3, 1056--67.
• 26: Nayar, S. K., Nakagawa, Y. 1990. Shape from focus: An effective approach for rough surfaces. Proc. International Conference on Robotics and Automation (ICRA), Vol. 2, 218--225.
• 27: Shree K. Nayar , Srinivasa G. Narasimhan, Assorted Pixels: Multi-sampled Imaging with Structural Models, Proceedings of the 7th European Conference on Computer Vision-Part IV, p.636-652, May 28-31, 2002
• 28: Ren Ng, Fourier slice photography, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
• 29: Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M., Hanrahan, P. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Stanford Tech Report CTSR 2005-02.
• 30: Ng, R. 2006. Digital Light Field Photography. PhD dissertation, Stanford University.
• 31: Noguchi, M., Nayar, S. 1994. Microscopic shape from focus using active illumination. Proc. IAPR International Conference on Pattern Recognition (ICPR), Vol. A, 147--152.
• 32: Okoshi, T. 1976. Three-Dimensional Imaging Techniques. Academic Press.
• 33: Piller, H. 1977. Microscope Photometry. Springer-Verlag.
• 34: Pluta, M. 1988. Advanced Light Microscopy (in 3 volumes). Elsevier.
• 35: Yoav Y. Schechner , Nahum Kiryati, The Optimal Axial Interval in Estimating Depth from Defocus, Proceedings of the International Conference on Computer Vision-Volume 2, p.843, September 20-25, 1999
• 36: Yoav Y. Schechner , Nahum Kiryati, Depth from Defocus vs. Stereo: How Different Really Are They?, International Journal of Computer Vision, v.39 n.2, p.141-162, Sept. 2000 [doi>10.1023/A:1008175127327 ]
• 37: Yoav Y. Schechner , Nahum Kiryati , Ronen Basri, Separation of Transparent Layers using Focus, International Journal of Computer Vision, v.39 n.1, p.25-39, Aug. 2000 [doi>10.1023/A:1008166017466 ]
• 38: Schechner, Y., Nayar, S. 2001. Generalized Mosaicing. Proc. ICCV.
• 39: Shah, U. B., Nayar, S. K. 1992. Extracting 3-D structure and focused images using an optical microscope. Proc. IEEE Symposium on Computer-Based Medical Systems.
• 40: Streibl, N. 1984. Depth transfer by an imaging system. Optica Acta 31, 11, 1233--1241.
• 41: Streibl, N. 1985. Three-dimensional imaging by a microscope. J. Opt. Soc. Am. A 2, 2, 121--127.
• 42: Jason R. Swedlow , John W. Sedat , David A. Agard, Deconvolution in optical microscopy, Deconvolution of images and spectra (2nd ed.), Academic Press, Inc., Orlando, FL, 1996
• 43: Vaibhav Vaish , Gaurav Garg , Eino-Ville Talvala , Emilio Antunez , Bennett Wilburn , Mark Horowitz , Marc Levoy, Synthetic Aperture Focusing using a Shear-Warp Factorization of the Viewing Transform, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, p.129, June 20-26, 2005 [doi>10.1109/CVPR.2005.537]
• 44: Weinstein, R. S., Descour, M. R., et al. 2004. An array microscope for ultrarapid virtual slide processing and telepathology. Design, fabrication, and validation study. Human Pathology 35, 11, 1303--1314.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture

Keywords:
deconvolution, light fields, microscopy, synthetic aperture, tomography, volume rendering