Image-based 3D photography using opacity hulls

ABSTRACT

We have built a system for acquiring and displaying high quality graphical models of objects that are impossible to scan with traditional scanners. Our system can acquire highly specular and fuzzy materials, such as fur and feathers. The hardware set-up consists of a turntable, two plasma displays, an array of cameras, and a rotating array of directional lights. We use multi-background matting techniques to acquire alpha mattes of the object from multiple viewpoints. The alpha mattes are used to construct an opacity hull. The opacity hull is a new shape representation, defined as the visual hull of the object with view-dependent opacity. It enables visualization of complex object silhouettes and seamless blending of objects into new environments. Our system also supports relighting of objects with arbitrary appearance using surface reflectance fields, a purely image-based appearance representation. Our system is the first to acquire and render surface reflectance fields under varying illumination from arbitrary viewpoints. We have built three generations of digitizers with increasing sophistication. In this paper, we present our results from digitizing hundreds of models.

References

1. BEARDSLEY, P. 2002. Calibration of stereo cameras for a turntable 3d scanner. Tech. rep., MERL. TR 2002/20.
2. BERNARDINI, F., MARTIN, I., AND RUSHMEIER, H. 2001. High-quality texture reconstruction from multiple scans. IEEE Trans. on Vis. and Comp. Graph. 7, 4 (Oct.-Dec.), 318-332.
3. BUEHLER, C., BOSSE, M., MCMILLAN, L., GORTLER, S., AND COHEN, M. 2001. Unstructured lumigraph rendering. In Computer Graphics, SIGGRAPH 2001 Proceedings, 425-432.
4. CHEN, S. E., AND WILLIAMS, L. 1993. View interpolation for image synthesis. In Computer Graphics, SIGGRAPH 93 Proceedings, 279-288.
5. CHEN, W.-C., GRZESZCZUK, R., AND BOUGUET, J.-Y. 2002. Light field mapping: Efficient representation and hardware rendering of surface light fields. In Computer Graphics, To appear in the SIGGRAPH 2002 Proceedings.
6. CURLESS, B., AND LEVOY, M. 1996. A volumetric method for building complex models from range images. In Computer Graphics, SIGGRAPH 96 Proceedings, 303-312.
7. DEBEVEC, P., AND MALIK, J. 1997. Recovering high dynamic range radiance maps from photographs. In Computer Graphics, SIGGRAPH 97 Proceedings, 369-378.
8. DEBEVEC, P., TAYLOR, C., AND MALIK, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Computer Graphics, SIGGRAPH 96 Proceedings, 11-20.
9. DEBEVEC, P., YU, Y., AND BORSHUKOV, G. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Proceedings of the 9th Eurographics Workshop on Rendering, 105-116.
10. DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H.-P., SAROKIN, W., AND SAGAR, M. 2000. Acquiring the reflectance field of a human face. In Computer Graphics, SIGGRAPH 2000 Proceedings, 145-156.
11. GEORGHIADES, A., BELHUMEUR, P., AND KRIEGMAN, D. 1999. Illumination-based image synthesis: Creating novel images of human faces under differing pose and lighting. In IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes, 47-54.
12. GORTLER, S., GRZESZCZUK, R., SZELISKI, R., AND COHEN, M. 1996. The lumigraph. In Computer Graphics, SIGGRAPH 96 Proceedings, 43-54.
13. HAWKINS, T., COHEN, J., AND DEBEVEC, P. 2001. A photometric approach to digitizing cultural artifacts. In 2nd International Symposium on Virtual Reality, Archaeology, and Cultural Heritage.
14. KOUDELKA, M., BELHUMEUR, P., MAGDA, S., AND KRIEGMAN, D. 2001. Image-based modeling and rendering of surfaces with arbitrary brdfs. In Proc. of Computer Vision and Pattern Recognition, in press.
15. LAURENTINI, A. 1994. The visual hull concept for silhouette-based image understanding. PAMI 16, 2 (February), 150-162.
16. LENGYEL, J., PRAUN, E., FINKELSTEIN, A., AND HOPPE, H. 2001. Real-time fur over arbitrary surfaces. In Symposium on Interactive 3D Graphics, 227-232.
17. LENSCH, H., KAUTZ, J., GOESELE, M., HEIDRICH, W., AND SEIDEL, H.-P. 2001. Image-based reconstruction of spatially varying materials. In Proceedings of the 12th Eurographics Workshop on Rendering.
18. LEVOY, M., AND HANRAHAN, P. 1996. Light field rendering. In Computer Graphics, SIGGRAPH 96 Proceedings, 31-42.
19. LEVOY, M., AND WHITTED, T. 1985. The use of points as display primitives. Tech. Rep. TR 85-022, The University of North Carolina at Chapel Hill, Department of Computer Science.
20. LEVOY, M., PULLI, K., CURLESS, B., RUSINKIEWICZ, S., KOLLER, D., PEREIRA, L., GINZTON, M., ANDERSON, S., DAVIS, J., GINSBERG, J., SHADE, J., AND FULK, D. 2000. The digital michelangelo project: 3d scanning of large statues. In Computer Graphics, SIGGRAPH 2000 Proceedings, 131-144.
21. MALZBENDER, T., GELB, D., AND WOLTERS, H. 2001. Polynomial texture maps. In Computer Graphics, SIGGRAPH 2001 Proceedings, 519-528.
22. MARSCHNER, S., WESTIN, S., LAFORTUNE, E., TORRANCE, K., AND GREENBERG, D. 1999. Image-based brdf measurement including human skin. In Proceedings of the 10th Eurographics Workshop on Rendering, 139-152.
23. MATUSIK, W., BUEHLER, C., RASKAR, R., GORTLER, S., AND MCMILLAN, L. 2000. Image-based visual hulls. In Computer Graphics, SIGGRAPH 2000 Proceedings, 369-374.
24. MCMILLAN, L., AND BISHOP, G. 1995. Plenoptic modeling: An image-based rendering system. In Computer Graphics, SIGGRAPH 95 Proceedings, 39-46.
25. MILLER, G., RUBIN, S., AND PONCELEON, D. 1998. Lazy decompression of surface light fields for precomputed global illumination. In Proceedings of the 9th Eurographics Workshop on Rendering, 281-292.
26. NISHINO, K., SATO, Y., AND IKEUCHI, K. 1999. Appearance compression and synthesis based on 3d model for mixed reality. In Proceedings of IEEE ICCV '99, 38-45.
27. NISHINO, K., SATO, Y., AND IKEUCHI, K. 1999. Eigen-texture method: Appearance compression based on 3d model. In Proc. of Computer Vision and Pattern Recognition, 618-624.
28. PFISTER, H., ZWICKER, M., VAN BAAR, J., AND GROSS, M. 2000. Surfels: Surface elements as rendering primitives. In Computer Graphics, SIGGRAPH 2000 Proceedings, 335-342.
29. PULLI, K., COHEN, M., DUCHAMP, T., HOPPE, H., SHAPIRO, L., AND STUETZLE, W. 1997. View-based rendering: Visualizing real objects from scanned range and color data. In Eurographics Rendering Workshop 1997, 23-34.
30. RAMAMOORTHI, R., AND HANRAHAN, P. 2001. A signal-processing framework for inverse rendering. In Computer Graphics, SIGGRAPH 2001 Proceedings, 117-128.
31. RUSHMEIER, H., BERNARDINI, F., MITTLEMAN, J., AND TAUBIN, G. 1998. Acquiring input for rendering at appropriate levels of detail: Digitizing a pietà. In Proceedings of the 9th Eurographics Workshop on Rendering, 81-92.
32. RUSINKIEWICZ, S., AND LEVOY, M. 2000. Qsplat: A multiresolution point rendering system for large meshes. In Computer Graphics, SIGGRAPH 2000 Proceedings, 343-352.
33. SANDER, P., GU, X., GORTLER, S., HOPPE, H., AND SNYDER, J. 2000. Silhouette clipping. In Computer Graphics, SIGGRAPH 2000 Proceedings, 327-334.
34. SATO, Y., WHEELER, M. D., AND IKEUCHI, K. 1997. Object shape and reflectance modeling from observation. In Computer Graphics, SIGGRAPH 97 Proceedings, 379-387.
35. SMITH, A. R., AND BLINN, J. F. 1996. Blue screen matting. In Computer Graphics, vol. 30 of SIGGRAPH 96 Proceedings, 259-268.
36. TURK, G., AND LEVOY, M. 1994. Zippered polygon meshes from range images. In Computer Graphics, SIGGRAPH 94 Proceedings, 311-318.
37. UDUPA, J., AND ODHNER, D. 1993. Shell rendering. IEEE Computer Graphics & Applications 13, 6 (Nov.), 58-67.
38. WOOD, D., AZUMA, D., ALDINGER, K., CURLESS, B., DUCHAMP, T., SALESIN, D., AND STUETZLE, W. 2000. Surface light fields for 3d photography. In Computer Graphics, SIGGRAPH 2000 Proceedings, 287-296.
39. YU, Y., DEBEVEC, P., MALIK, J., AND HAWKINS, T. 1999. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Computer Graphics, SIGGRAPH 99 Proceedings, 215-224.
40. ZONGKER, D., WERNER, D., CURLESS, B., AND SALESIN, D. 1999. Environment matting and compositing. In Computer Graphics, SIGGRAPH 99 Proceedings, 205-214.
41. ZWICKER, M., PFISTER., H., BAAR, J. V., AND GROSS, M. 2001. Surface splatting. In Computer Graphics, SIGGRAPH 2001 Proceedings, 371-378.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Document scanning

      • Graphics recognition and interpretation

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation

    • Image manipulation

    • Rendering