Methods and metrics for cold-start recommendations

ABSTRACT

We have developed a method for recommending items that combines content and collaborative data under a single probabilistic framework. We benchmark our algorithm against a naïve Bayes classifier on the cold-start problem, where we wish to recommend items that no one in the community has yet rated. We systematically explore three testing methodologies using a publicly available data set, and explain how these methods apply to specific real-world applications. We advocate heuristic recommenders when benchmarking to give competent baseline performance. We introduce a new performance metric, the CROC curve, and demonstrate empirically that the various components of our testing strategy combine to obtain deeper understanding of the performance characteristics of recommender systems. Though the emphasis of our testing is on cold-start recommending, our methods for recommending and evaluation are general.

References

1. C. Basu, H. Hirsh, and W. Cohen. Recommendation as classification: Using social and content-based information in recommendation. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pages 714--720, 1998.
2. D. Billsus and M. J. Pazzani. Learning collaborative information filters. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 46--54, 1998.
3. J. S. Breese, D. Heckerman, and C. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pages 43--52, 1998.
4. M. Claypool, A. Gokhale, and T. Miranda. Combining content-based and collaborative filters in an online newspaper. In Proceedings of the ACM SIGIR Workshop on Recommender Systems---Implementation and Evaluation, 1999.
5. W. W. Cohen, R. E. Schapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243--270, 1999.
6. M. K. Condliff, D. D. Lewis, D. Madigan, and C. Posse. Bayesian mixed-effect models for recommender systems. In ACM SIGIR '99 Workshop on Recommender Systems: Algorithms and Evaluation, 1999.
7. Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer. An efficient boosting algorithm for combining preferences. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 170--178, 1998.
8. N. Good, J. B. Schafer, J. A. Konstan, A. Borchers, B. M. Sarwar, J. L. Herlocker, and J. Riedl. Combining collaborative filtering with personal agents for better recommendations. In Proceedings of the Sixteenth National Conference on Artificial Intelligence, pages 439--446, 1999.
9. H. Guo. Soap: Live recommendations through social agents. In Fifth DELOS Workshop on Filtering and Collaborative Filtering, Budapest, 1997.
10. D. Heckerman, D. M. Chickering, C. Meek, R. Rounthwaite, and C. Kadie. Dependency networks for collaborative filtering and data visualization. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, pages 264--273, 2000.
11. J. L. Herlocker, J. A. Konstan, A. Borchers, and J. Riedl. An algorithmic framework for performing collaborative filtering. In Proceedings of the Conference on Research and Development in Information Retrieval, 1999.
12. W. Hill, L. Stead, M. Rosenstein, and G. Furnas. Recommending and evaluating choices in a virtual community of use. In Proceedings of the Conference on Human Factors in Computing Systems, pages 194--201, 1995.
13. T. Hofmann. Probabilistic latent semantic indexing. In 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 50--57, 1999.
14. T. Hofmann and J. Puzicha. Latent class models for collaborative filtering. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, pages 688--693, 1999.
15. J. A. Konstan, B. N. Miller, D. Maltz, J. L. Herlocker, L. R. Gordon, and J. Riedl. GroupLens: Applying collaborative filtering to Usenet news. Communications of the ACM, 40(3):77--87, 1997.
16. P. Melville, R. J. Mooney, and R. Nagarajan. Content-boosted collaborative filtering. In Proceedings of the 2001 SIGIR Workshop on Recommender Systems, 2001.
17. R. J. Mooney and L. Roy. Content-based book recommending using learning for text categorization. In Proceedings of the Fifth ACM Conference on Digital Libraries, pages 195--204, 2000.
18. A. Nakamura and N. Abe. Collaborative filtering using weighted majority prediction algorithms. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 395--403, 1998.
19. D. M. Pennock, E. Horvitz, and C. L. Giles. Social choice theory and recommender systems: Analysis of the axiomatic foundations of collaborative filtering. In Proceedings of the Seventeenth National Conference on Artificial Intelligence, pages 729--734, 2000.
20. D. M. Pennock, E. Horvitz, S. Lawrence, and C. L. Giles. Collaborative filtering by personality diagnosis: A hybrid memory- and model-based approach. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, pages 473--480, 2000.
21. A. Popescul, L. H. Ungar, D. M. Pennock, and S. Lawrence. Probabilistic models for unified collaborative and content-based recommendation in sparse-data environments. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, 2001.
22. P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl. GroupLens: An open architecture for collaborative filtering of netnews. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, pages 175--186, 1994.
23. P. Resnick and H. R. Varian. Recommender systems. Communications of the ACM, 40(3):56--58, 1997.
24. G. Salton and M. McGill. Introduction to Modern Information Retrieval. McGraw Hill, 1983.
25. B. M. Sarwar, G. Karypis, J. A. Konstan, and J. T. Riedl. Analysis of recommender algorithms for e-commerce. In Proceedings of the 2nd ACM Conference on Electronic Commerce, pages 158--167, 2000.
26. B. M. Sarwar, G. Karypis, J. A. Konstan, and J. T. Riedl. Application of dimensionality reduction in recommender system---A case study. In ACM WebKDD Web Mining for E-Commerce Workshop, 2000.
27. B. M. Sarwar, G. Karypis, J. A. Konstan, and J. T. Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the Tenth International World Wide Web Conference, pages 285--295, 2001.
28. A. I. Schein, A. Popescul, and L. H. Ungar. PennAspect: A two-way aspect model implementation. Technical Report MS-CIS-01-25, Department of Computer and Information Science, The University of Pennsylvania, 2001.
29. A. I. Schein, A. Popescul, L. H. Ungar, and D. M.Pennock. Generative models for cold-start recommendations. In Proceedings of the 2001 SIGIR Workshop on Recommender Systems, 2001.
30. U. Shardanand and P. Maes. Social information filtering: Algorithms for automating 'word of mouth'. In Proceedings of the Conference on Human Factors in Computing Systems, pages 210--217, 1995.
31. L. H. Ungar and D. P. Foster. Clustering methods for collaborative filtering. In Workshop on Recommendation Systems at the Fifteenth National Conference on Artificial Intelligence, 1998.

Index Terms

• Computing methodologies

  • Machine learning

• Mathematics of computing

  • Information theory

  • Probability and statistics

    • Statistical paradigms

      • Contingency table analysis