Visio-lization: generating novel facial images

ABSTRACT

Our goal is to generate novel realistic images of faces using a model trained from real examples. This model consists of two components: First we consider face images as samples from a texture with spatially varying statistics and describe this texture with a local non-parametric model. Second, we learn a parametric global model of all of the pixel values. To generate realistic faces, we combine the strengths of both approaches and condition the local non-parametric model on the global parametric model. We demonstrate that with appropriate choice of local and global models it is possible to reliably generate new realistic face images that do not correspond to any individual in the training data. We extend the model to cope with considerable intra-class variation (pose and illumination). Finally, we apply our model to editing real facial images: we demonstrate image in-painting, interactive techniques for improving synthesized images and modifying facial expressions.

References

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 3, 294--302.
2. Ashikhmin, M. 2001. Synthesizing natural textures. In Proc. ACM Symposium on Interactive 3D Graphics, 217--226.
3. Bishop, C. 2006. Pattern Recognition and Machine Learning. Springer.
4. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: automatically replacing faces in photographs. ACM Trans. Graph. 27, 3, 1--8.
5. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM SIGGRAPH 99, 187--194.
6. Blanz, V., Albrecht, I., Haber, J., and Seidel, H.-P. 2006. Creating face models from vague mental images. Computer Graphics Forum 25, 3, 645--654.
7. Brand, M., and Pletscher, P. 2008. A conditional random field for photo editing. In Proceedings of CVPR, 187--194.
8. Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: Driving visual speech with audio. In Proceedings of ACM SIGGRAPH 97, 353--360.
9. Dedeoglu, G., Kanade, T., and August, J. 2004. Highzoom video hallucination by exploiting spatio-temporal regularities. In Proceedings of CVPR, 151--158.
10. Dempster, A. P., Laird, N. M., and Rubin, D. B. 1977. Maximum likelihood for incomplete data via the EM algorithm. Journal of the Royal Statistical Society 39 (B), 1, 1--38.
11. Diakopoulos, N., Essa, I., and Jain, R. 2004. Content based image synthesis. In CIVR 04, 299--307.
12. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proceedings of ACM SIGGRAPH, 341--346.
13. Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In Proceedings of ICCV, vol. 2, 1033--1038.
14. Ezzat, T., Geiger, G., and Poggio, T. 2002. Trainable videorealistic speech animation. In Proceedings of ACM SIGGRAPH 2002, 388--398.
15. Ghahramani, Z., and Hinton, G. E. 1997. The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, Dept. of Computer Science, University of Toronto, Canada.
16. Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, 3, 4:1--4:7.
17. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics 22, 3, 277--286.
18. Lalonde, J., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, 3, 3:1--3:10.
19. Liu, W., Lin, D., and Tang, X. 2005. Hallucinating faces: Tensorpatch super-resolution and coupled residue compensation. In Proceedings of CVPR, 478--484.
20. Liu, C., Shum, H., and Freeman, W. 2007. Face hallucination: theory and practice. International Journal of Computer Vision 75, 1, 115--134.
21. Messer, K., Matas, J., Kittler, J., Luettin, J., and Maitre, G. 1999. XM2VTSbd: The extended MTVTS database. In Proceedings 2nd Conference on Audio and Videobase Biometric Personal Verification (AVBPA99), 72--77.
22. Nguyen, M., Lalonde, J., Efros, A., and La Torre, F. D. 2008. Image-based shaving. Computer Graphics Forum (Eurographics) 27, 2, 627--635.
23. Perez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics (Proc. SIGGRAPH) 22, 3, 313--318.
24. Prince, S., Elder, J., Warrell, J., and Felisberti, F. 2008. Tied factor analysis for face recognition across large pose differences. IEEE Pattern Recognition and Machine Intelligence 30, 6, 970--984.
25. Tenenbaum, J., and Freeman, W. 2000. Separating style and content with bilinear models. Neural Computation 12, 6, 1247--1283.
26. Turk, M. A., and Pentland, A. P. 1991. Face recognition using eigenfaces. In Proceedings of CVPR, 586--591.
27. Vlasic, D., Brand, M., Pfister, H., and Popovic, J. 2005. Face transfer with multiliner models. ACM Transactions on Graphics (Proc. SIGGRAPH) 24, 3, 426--433.
28. Wei, L., and Levoy, M. 2000. Fast texture synthesis using tree-structured vector quantization. In Proceedings of ACM SIGGRAPH 2000, 479--488.
29. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics (Proc. SIGGRAPH) 25, 3, 1013--1024.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Computer graphics