The support vector decomposition machine

ABSTRACT

In machine learning problems with tens of thousands of features and only dozens or hundreds of independent training examples, dimensionality reduction is essential for good learning performance. In previous work, many researchers have treated the learning problem in two separate phases: first use an algorithm such as singular value decomposition to reduce the dimensionality of the data set, and then use a classification algorithm such as naïve Bayes or support vector machines to learn a classifier. We demonstrate that it is possible to combine the two goals of dimensionality reduction and classification into a single learning objective, and present a novel and efficient algorithm which optimizes this objective directly. We present experimental results in fMRI analysis which show that we can achieve better learning performance and lower-dimensional representations than two-phase approaches can.

REFERENCES

• 1: Stephen Boyd , Lieven Vandenberghe, Convex Optimization, Cambridge University Press, New York, NY, 2004
• 2: Rich Caruana, Multitask Learning, Machine Learning, v.28 n.1, p.41-75, July 1997 [doi>10.1023/A:1007379606734 ]
• 3: Chang, C.-C., & Lin, C.-J. (2001). LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.
• 4: Globerson, A., & Roweis, S. (2005). Metric learning by collapsing classes. Advances in Neural Information Processing Systems.
• 5: Isabelle Guyon , Jason Weston , Stephen Barnhill , Vladimir Vapnik, Gene Selection for Cancer Classification using Support Vector Machines, Machine Learning, v.46 n.1-3, p.389-422, 2002
• 6: Hanson, S. J., Matsuka, T., & Haxby, J. V. (2004). Combinatorial codes in ventral temporal lobe for object recognition: Haxby(2001) revisited: is there a 'face' area? Neuroimage, 23.
• 7: Hastie, T., Tibshirani, R., & Friedman, J. H. (2001). The elements of statistical learning: data mining, inference and prediction. Springer-Verlag.
• 8: Kanwisher, N. (2003). The ventral visual object pathway in humans: Evidence from fmri. In L. Chalupa and J. Werner (Eds.), The visual neurosciences. MIT Press.
• 9: Tom M. Mitchell , Rebecca Hutchinson , Radu S. Niculescu , Francisco Pereira , Xuerui Wang , Marcel Just , Sharlene Newman, Learning to Decode Cognitive States from Brain Images, Machine Learning, v.57 n.1-2, p.145-175, October-November 2004 [doi>10.1023/B:MACH.0000035475.85309.1b]
• 10: Pereira, F., Mitchell, T., Mason, R., Just, M., & Kriegeskorte, N. (2006). "spatial searchlights for feature selection and classification of functional mri data". to appear in the proceedings of the 12th Conference on Human Brain Mapping.
• 11: Nathan Srebro , Tommi S. Jaakkola, Learning with matrix factorizations, Massachusetts Institute of Technology, Cambridge, MA, 2004
• 12: Weinberger, K. Q., Blitzer, J., & Saul, L. K. (2005). Distance metric learning for large margin nearest neighbour classification. Advances in Neural Information Processing Systems.
• 13: Xing, E. P., Ng, A. Y., Jordan, M. I., & Russell, S. (2002). Distance metric learning, with application to clustering with side-information. Advances in Neural Information Processing Systems.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.1 Models
          Subjects: Neural nets

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Connectionism and neural nets
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology

General Terms:
Algorithms