Do clicks measure recommendation relevancy?: an empirical user study

ABSTRACT

Evaluation have been an important subject since the early days of recommender systems. In online test, the click-through rate (CTR) is often adopted as the metric. However, recommended items with higher CTR does not imply higher relevance of two items since factors like item popularity or item serendipity may influence user's click behavior. We argue that the relevance of recommendation system is also desirable in many real applications. Here relevant means relevance in a human perceptible way. Relevant recommendations not only increase the users' trust to the system but are extremely useful for the vast number of anonymous user as their recommendations may only be made based on the current item. In this paper, we empirically examine the relation between the relevance of recommendations and the corresponding CTR with a few representative ItemCF algorithms through online data from a TV show/movie website, Hulu. Experiments show that algorithms with higher overall CTR may not correspond to higher relevance. Thus CTR may not be the optimal metric for online evaluation of recommender systems if producing relevant recommendations is of importance.

References

1. }}R. M. Bell and Y. Koren. Lessons from the netflix prize challenge. SIGKDD Explor. Newsl., 9(2):75--79, 2007.
2. }}R. L. Cilibrasi and P. M. B. Vitanyi. The google similarity distance. IEEE Trans. on Knowl. and Data Eng., 19(3):370--383, 2007.
3. }}D. Cosley, S. K. Lam, I. Albert, J. A. Konstan, and J. Riedl. Is seeing believing?: how recommender system interfaces affect users' opinions. In CHI '03, pages 585--592, New York, NY, USA, 2003. ACM.
4. }}M. Deshpande and G. Karypis. Item-based top-n recommendation algorithms. ACM Trans. Inf. Syst., 22(1):143--177, 2004.
5. }}J. Herlocker, J. Konstan, L. Terveen, and J. Riedl. Evaluating Collaborative Filtering Recommender Systems. ACM Transactions on Information Systems, 22:5--53, 2004.
6. }}J. Konstan, B. Miller, D. Maltz, J. Herlocker, L. Gordon, and J. Riedl. GroupLens: Applying Collaborative Filtering to Usenet News. Communications of the ACM, 40:77--87, 1997.
7. }}G. Linden, B. Smith, and J. York. Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Computing, 7(1):76--80, 2003.
8. }}J. Liu, P. Dolan, and E. R. Pedersen. Personalized news recommendation based on click behavior. In IUI, pages 31--40, 2010.
9. }}S. McNee, J. Riedl, and J. Konstan. Being Accurate is Not Enough: How Accuracy Metrics Have Hurt Recommender Systems. In Extended Abstracts CHI'06, Montreal, Canada, April 2006.
10. }}M. Pazzani and D. Billsus. Content-Based Recommendation Systems. The Adaptive Web, 4321:325--341, 2007.
11. }}M. Weimer, A. Karatzoglou, Q. V. Le, and A. J. Smola. Cofi rank - maximum margin matrix factorization for collaborative ranking. In NIPS, 2007.
12. }}M. Zhang and N. Hurley. Avoiding monotony: improving the diversity of recommendation lists. In RecSys '08: Proceedings of the 2008 ACM conference on Recommender systems, pages 123--130, New York, NY, USA, 2008. ACM.

Index Terms

• Information systems

  • Information retrieval

    • Retrieval tasks and goals

      • Document filtering

      • Information extraction