skip to main content
10.1145/3109859.3109911acmconferencesArticle/Chapter ViewAbstractPublication PagesrecsysConference Proceedingsconference-collections
research-article
Open Access

Folding: Why Good Models Sometimes Make Spurious Recommendations

Published:27 August 2017Publication History

ABSTRACT

In recommender systems based on low-rank factorization of a partially observed user-item matrix, a common phenomenon that plagues many otherwise effective models is the interleaving of good and spurious recommendations in the top-K results. A single spurious recommendation can dramatically impact the perceived quality of a recommender system. Spurious recommendations do not result in serendipitous discoveries but rather cognitive dissonance. In this work, we investigate folding, a major contributing factor to spurious recommendations. Folding refers to the unintentional overlap of disparate groups of users and items in the low-rank embedding vector space, induced by improper handling of missing data. We formally define a metric that quantifies the severity of folding in a trained system, to assist in diagnosing its potential to make inappropriate recommendations. The folding metric complements existing information retrieval metrics that focus on the number of good recommendations and their ranks but ignore the impact of undesired recommendations. We motivate the folding metric definition on synthetic data and evaluate its effectiveness on both synthetic and real world datasets. In studying the relationship between the folding metric and other characteristics of recommender systems, we observe that optimizing for goodness metrics can lead to high folding and thus more spurious recommendations.

References

  1. Panagiotis Adamopoulos and Alexander Tuzhilin. 2015. On unexpectedness in recommender systems: Or how to better expect the unexpected. ACM Transactions on Intelligent Systems and Technology (2015). Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Konstantina Christakopoulou and Arindam Banerjee. 2015. Collaborative Ranking with a Push at the Top. International World Wide Web Conference (WWW) (2015). Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010. Performance of recommender algorithms on top-n recommendation tasks. In RecSys. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In RecSys. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Quanquan Gu, Jie Zhou, and Chris Ding. 2010. Collaborative filtering: Weighted nonnegative matrix factorization incorporating user and item graphs. In International Conference on Data Mining. SIAM.Google ScholarGoogle ScholarCross RefCross Ref
  6. Asela Gunawardana and Guy Shani. 2009. A survey of accuracy evaluation metrics of recommendation tasks. Journal of Machine Learning Research (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (2015). Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Elad Hazan, Roi Livni, and Yishay Mansour. 2015. Classification with Low Rank and Missing Data. In ICML. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. 2000. Explaining collaborative filtering recommendations. In CSCW. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John T. Riedl. 2004. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS) (2004). Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. José Miguel Hernández-Lobato, Neil Houlsby, and Zoubin Ghahramani. 2014. Probabilistic Matrix Factorization with Non-random Missing Data. In ICML. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In International Conference on Data Mining (ICDM). IEEE. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. Association for Computational Linguistics (2015).Google ScholarGoogle Scholar
  14. Noriaki Kawamae. 2010. Serendipitous recommendations via innovators. In SIGIR. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction (2012). Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Joseph A. Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User Modeling and User-Adapted Interaction (2012). Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. IEEE Computer (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. 2016. Modeling user exposure in recommendation. In International Conference on World Wide Web (WWW). Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Roderick J. A. Little and Donald B. Rubin. 2014. Statistical analysis with missing data. John Wiley & Sons.Google ScholarGoogle Scholar
  20. Benjamin M. Marlin and Richard S. Zemel. 2009. Collaborative prediction and ranking with non-random missing data. In RecSys. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Benjamin M. Marlin, Richard S. Zemel, Sam Roweis, and Malcolm Slaney. 2007. Collaborative Filtering and the Missing at Random Assumption. In Uncertainty in Artificial Intelligence (UAI). AUAI Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Making recommendations better: an analytic model for human-recommender interaction. In CHI extended abstracts on Human factors in computing systems. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In NIPS. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Rong Pan and Martin Scholz. 2009. Mind the Gaps: Weighting the Unknown in Large-scale One-class Collaborative Filtering. In KDD. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Rong Pan, Yunhong Zhou, Bin Cao, Nathan N. Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. 2008. One-Class Collaborative Filtering. In ICDM. IEEE Computer Society.Google ScholarGoogle Scholar
  26. Bruno Pradel, Nicolas Usunier, and Patrick Gallinari. 2012. Ranking with non-random missing ratings: influence of popularity and positivity on evaluation metrics. In RecSys. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. arXiv preprint arXiv:1602.05352 (2016). Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Guy Shani and Asela Gunawardana. 2011. Evaluating recommendation systems. In Recommender systems handbook. Springer, 257--297.Google ScholarGoogle Scholar
  29. Harald Steck. 2010. Training and testing of recommender systems on data missing not at random. In KDD. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Hsiang-Fu Yu, Mikhail Bilenko, and Chih-Jen Lin. 2017. Selection of Negative Samples for One-class Matrix Factorization. In SDM. SIAM.Google ScholarGoogle Scholar
  31. Yin Zheng, Bangsheng Tang, Wenkui Ding, and Hanning Zhou. 2016. A Neural Autoregressive Approach to Collaborative Filtering. In ICML. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Folding: Why Good Models Sometimes Make Spurious Recommendations

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader