Collaborative filtering on skewed datasets

ABSTRACT

Many real life datasets have skewed distributions of events when the probability of observing few events far exceeds the others. In this paper, we observed that in skewed datasets the state of the art collaborative filtering methods perform worse than a simple probabilistic model. Our test bench includes a real ad click stream dataset which is naturally skewed. The same conclusion is obtained even from the popular movie rating dataset when we pose a binary prediction problem of whether a user will give maximum rating to a movie or not.

REFERENCES

• 1: Adomavicius, G. and Tuzhilin, A. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Transactions on Knowledge and Data Engineering, 2005, Vol 17, 734--749.
• 2: Das, A. Datar, M. Garg, A. and Rajaram, S. Google News Personalization: Online Collaborative Filtering. Proceedings of the 16th international conference on World Wide Web, Banff, Canada, 2007.
• 3: Hofmann, T. Latent Semantic Models for Collaborative Filtering. ACM Transactions on Information Systems (TOIS), 2004, Vol 22, 89--115.
• 4: Hsu, C. N. Chung, H. H. and Huang, H. S. Mining Skewed and Sparse Transaction Data for Personalized Shopping Recommendation. Machine Learning, 2004, Vol 57, 35--59.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Information filtering

General Terms:
Algorithms, Experimentation

Keywords:
collaborative filtering, plsa, skewed dataset