Robust probabilistic projections

ABSTRACT

Principal components and canonical correlations are at the root of many exploratory data mining techniques and provide standard pre-processing tools in machine learning. Lately, probabilistic reformulations of these methods have been proposed (Roweis, 1998; Tipping & Bishop, 1999b; Bach & Jordan, 2005). They are based on a Gaussian density model and are therefore, like their non-probabilistic counterpart, very sensitive to atypical observations. In this paper, we introduce robust probabilistic principal component analysis and robust probabilistic canonical correlation analysis. Both are based on a Student-t density model. The resulting probabilistic reformulations are more suitable in practice as they handle outliers in a natural way. We compute maximum likelihood estimates of the parameters by means of the EM algorithm.

REFERENCES

• 1: Archambeau, C. (2005). Probabilistic models in noisy environments and their application to a visual prosthesis for the blind. Doctoral dissertation, Université catholique de Louvain, Belgium.
• 2: Bach, F. R., & Jordan, M. I. (2005). A probabilistic interpretation of canoncial correlation analysis (Technical Report 688). Department of Statistics, University of California, Berkeley.
• 3: de la Torre, F., & Black, M. J. (2001). Robust principal component analysis for computer vision. Int. Conf. on Computer Vision (pp. 362--369).
• 4: Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society B, 39, 1--38.
• 5: Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24, 417--441.
• 6: Hotelling, H. (1936). Relations between two sets of variates. Biometrika, 28, 321--377.
• 7: Jolliffe, I. T. (1986). Principal component analysis. New York: Springer-Verlag.
• 8: Liu, C., & Rubin, D. B. (1995). ML estimation of the t distribution using EM and its extensions, ECM and ECME. Statistica Sinica, 5, 19--39.
• 9: Radford M. Neal , Geoffrey E. Hinton, A view of the EM algorithm that justifies incremental, sparse, and other variants, Learning in graphical models, MIT Press, Cambridge, MA, 1999
• 10: D. Peel , G. J. McLachlan, Robust mixture modelling using the t distribution, Statistics and Computing, v.10 n.4, p.339-348, October 2000 [doi>10.1023/A:1008981510081 ]
• 11: Sam Roweis, EM algorithms for PCA and SPCA, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.626-632, July 1998, Denver, Colorado, United States
• 12: Michael E. Tipping , Christopher M. Bishop, Mixtures of probabilistic principal component analyzers, Neural Computation, v.11 n.2, p.443-482, Feb. 15, 1999 [doi>10.1162/089976699300016728 ]
• 13: Tipping, M. E., & Bishop, C. M. (1999b). Probabilistic principal component analysis. Journal of the Royal Statistical Society B, 61, 611--622.
• 14: Verbeek, J., Roweis, S., & Vlassis, N. (2004). Nonlinear CCA and PCA by alignment of local models. Advances in Neural Information Processing Systems 16.
• 15: Xu, L., & Yuille, A. L. (1995). Robust principal component analysis by self-organizing rules based on statistical physics approach. IEEE Transactions on Neural Networks, 6, 131--143.

INDEX TERMS

Primary Classification:
  F. Theory of Computation
  F.1 COMPUTATION BY ABSTRACT DEVICES
      F.1.2 Modes of Computation
          Subjects: Probabilistic computation

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Correlation and regression analysis; Probabilistic algorithms (including Monte Carlo)

General Terms:
Theory