Abstract
Current music recommender systems typically act in a greedy manner by recommending songs with the highest user ratings. Greedy recommendation, however, is suboptimal over the long term: it does not actively gather information on user preferences and fails to recommend novel songs that are potentially interesting. A successful recommender system must balance the needs to explore user preferences and to exploit this information for recommendation. This article presents a new approach to music recommendation by formulating this exploration-exploitation trade-off as a reinforcement learning task. To learn user preferences, it uses a Bayesian model that accounts for both audio content and the novelty of recommendations. A piecewise-linear approximation to the model and a variational inference algorithm help to speed up Bayesian inference. One additional benefit of our approach is a single unified model for both music recommendation and playlist generation. We demonstrate the strong potential of the proposed approach with simulation results and a user study.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach
- S. Agrawal and N. Goyal. 2012. Analysis of thompson sampling for the multi-armed bandit problem. In Proceedings of the 25th Annual Conference on Learning Theory (COLT'12).Google Scholar
- N. Aizenberg, Y. Koren, and O. Somekh. 2012. Build your own music recommender by modeling internet radio streams. In Proceedings of the 21st International Conference on World Wide Web (WWW'12). ACM Press, New York, 1--10. Google ScholarDigital Library
- P. Auer. 2003. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res. 3, 397--422. Google ScholarDigital Library
- M. Braunhofer, M. Kaminskas, and F. Ricci. 2013. Location-aware music recommendation. Int. J. Multimedia Inf. Retr. 2, 1, 31--44.Google ScholarCross Ref
- R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16, 5, 1190--1208. Google ScholarDigital Library
- P. Cano, M. Koppenberger, and N. Wack. 2005. Content-based music audio recommendation. In Proceedings of the 13th Annual ACM International Conference on Multimedia (MM'05). ACM Press, New York, 211--212. Google ScholarDigital Library
- P. Castells, S. Vargas, and J. Wang. 2011. Novelty and diversity metrics for recommender systems: Choice, discovery and relevance. In Proceedings of the International Workshop on Diversity in Document Retrieval (DDR'11) at the 33rd European Conference on Information Retrieval (ECIR'11). 29--36.Google Scholar
- H. C. Chen and A. L. P. Chen. 2001. A music recommendation system based on music data grouping and user interests. In Proceedings of the 10th International Conference on Information and Knowledge Management (CIKM'01). ACM Press, New York, 231--238. Google ScholarDigital Library
- S. Chen, J. L. Moore, D. Turnbull, and T. Joachims. 2012. Playlist prediction via metric embedding. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'12). 714--722. Google ScholarDigital Library
- X. Chen, P. N. Bennett, K. Collins-Thompson, and E. Horvitz. 2013. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM'13). ACM Press, New York, 193--202. Google ScholarDigital Library
- Z. Cheng and J. Shen. 2014. Just-for-me: An adaptive personalization system for location-aware social music recommendation. In Proceedings of International Conference on Multimedia Retrieval (ICMR'14). 185. Google ScholarDigital Library
- C. Y. Chi, R. T. H. Tsai, J. Y. Lai, and J. Y. Jen Hsu. 2010. A reinforcement learning approach to emotion-based automatic playlist generation. In Proceedings of the International Conference on Technologies and Applications of Artificial Intelligence. IEEE Computer Society, 60--65. Google ScholarDigital Library
- H. Ebbinghaus. 1913. Memory: A Contribution to Experimental Psychology. Educational reprints. Teachers College, Columbia University.Google ScholarCross Ref
- D. Eck, P. Lamere, T. Bertin-Mahieux, and S. Green. 2007. Automatic generation of social tags for music recommendation. In Proceedings of the Neural Information Processing Systems Conference (NIPS'07). Vol. 20.Google Scholar
- N. Friedman and D. Koller. 2009. Probabilistic Graphical Models: Principles and Techniques 1st Ed. The MIT Press. Google ScholarDigital Library
- N. Golovin and E. Rahm. 2004. Reinforcement learning architecture for web recommendations. In Proceedings of the International Conference on Information Technology: Coding and Computing. Vol. 1, IEEE, 398--402. Google ScholarDigital Library
- A. Gunawardana and G. Shani. 2009. A survey of accuracy evaluation metrics of recommendation tasks. J. Mach. Learn. Res 10, 2935--2962. Google ScholarDigital Library
- N. Hariri, B. Mobasher, and R. Burke. 2012. Context-aware music recommendation based on latenttopic sequential patterns. In Proceedings of the 6th ACM Conference on Recommender Systems (RecSys'12). ACM Press, New York, 131--138. Google ScholarDigital Library
- T. Hastie, R. Tibshirani, and J. Friedman. 2009. The Elements of Statistical Learning. Springer.Google Scholar
- Y. Hu and M. Ogihara. 2011. Nextone player: A music recommendation system based on user behavior. In Proceedings of the 12th International Society for Music Information Retrieval Conference (ISMIR'11).Google Scholar
- T. Joachims, D. Freitag, and T. Mitchell. 1997. WebWatcher: A tour guide for the world wide web. In Proceedings of the 15th International Joint Conference on Artificial Intelligence (IJCAI'97). 770--777.Google Scholar
- M. Kaminskas, F. Ricci, and M. Schedl. 2013. Location-aware music recommendation using auto-tagging and hybrid matching. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys'13). ACM Press, New York, 17--24. Google ScholarDigital Library
- R. Karimi, C. Freudenthaler, A. Nanopoulos, and L. Schmidt-Thieme. 2011. Towards optimal active learning for matrix factorization in recommender systems. In Proceedings of the 23rd IEEE International Conference on Tools with Artificial Intelligence (ICTAI'11). 1069--1076. Google ScholarDigital Library
- E. Kaufmann, O. Cappé, and A. Garivier. 2012. On bayesian upper confidence bounds for bandit problems. J. Mach. Learn. Res. Proc. Track 22, 592--600.Google Scholar
- P. Knees and M. Schedl. 2013. A survey of music similarity and recommendation from music context data. ACM Trans. Multimedia Comput. Comm. Appl. 10, 1. Google ScholarDigital Library
- Y. Koren, R. Bell, and C. Volinsky. 2009. Matrix factorization techniques for recommender systems. Comput. 42, 8, 30--37. Google ScholarDigital Library
- N. Lathia, S. Hailes, L. Capra, and X. Amatriain. 2010. Temporal diversity in recommender systems. In Proceedings of the 33rd International ACM SIGIR Conference (SIGIR'10). ACM Press, New York, 210--217. Google ScholarDigital Library
- L. Li, W. Chu, J. Langford, and R. E. Schapire. 2012. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW'10). ACM Press, New York, 661--670. Google ScholarDigital Library
- L. Li, W. Chu, J. Langford, and X. Wang. 2011. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM'11). ACM Press, New York, 297--306. Google ScholarDigital Library
- E. Liebman and P. Stone. 2014. Dj-mc: A reinforcement-learning agent for music playlist recommendation. http://arxiv.org/pdf/1401.1880.pdfGoogle Scholar
- H. Liu, J. Hu, and M. Rauterberg. 2009. Music playlist recommendation based on user heartbeat and music preference. In Proceedings of the International Conference on Computer Technology and Development. Vol. 1, 545--549. Google ScholarDigital Library
- L. Liu, H. Xu, J. Xing, S. Liu, X. Zhou, and S. Yan. 2013. Wow! You are so beautiful today! In Proceedings of the 21st ACM International Conference on Multimedia (MM'13). 3--12. Google ScholarDigital Library
- B. Logan. 2002. Content-based playlist generation: Exploratory experiments. In Proceedings of the 3rd International Conference on Music Information Retrieval (ISMIR'02). 295--296.Google Scholar
- D. P. Mackinnon, M. S. Fritz, J. Williams, and C. M. Lockwood. 2007. Distribution of the product confidence limits for the indirect effect: Program prodclin. Behav. Res. Methods 39, 3, 384--389.Google ScholarCross Ref
- B. C. May, N. Korda, A. Lee, and D. S. Leslie. 2012. Optimistic bayesian sampling in contextual-bandit problems. J. Mach. Learn. Res. 13, 1, 2069--2106. Google ScholarDigital Library
- M. E. J. Newman. 2005. Power laws, pareto distributions and zipf's law. Contemp. Phys. 46, 5, 323--351.Google ScholarCross Ref
- A. V. D. Oord, S. Dieleman, and B. Schrauwen. 2013. Deep content-based music recommendation. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'13). 2643--2651.Google Scholar
- R. Salakhutdinov and A. Mnih. 2008. Bayesian probabilistic matrix factorization using markov chain monte carlo. In Proceedings of the 25th International Conference on Machine Learning (ICML'08). ACM Press, New York, 880--887. Google ScholarDigital Library
- M. Schedl and D. Schnitzer. 2014. Location-aware music artist recommendation. In Proceedings of the 20th International Conference on MultiMedia Modeling (MMM'14).Google Scholar
- G. Shani, D. Heckerman, and R. I. Brafman. 2005. An mdp-based recommender system. J. Mach. Learn. Res 6, 1265--1295. Google ScholarDigital Library
- J. Shen, X. S. Hua, and E. Sargin. 2013. Towards next generation multimedia recommendation systems. In Proceedings of the 21st ACM International Conference on Multimedia (MM'13). ACM Press, New York, 1109--1110. Google ScholarDigital Library
- J. Silva and L. Carin. 2012. Active learning for online bayesian matrix factorization. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'12). ACM Press, New York, 325--333. Google ScholarDigital Library
- Y. Song, S. Dixon, and M. Pearce. 2012. A survey of music recommendation systems and future perspectives. In Proceedings of the 9th International Symposium on Computer Music Modelling and Retrieval (CMMR'12).Google Scholar
- A. Srivihok and P. Sukonmanee. 2005. E-commerce intelligent agent: Personalization travel support agent using q learning. In Proceedings of the 7th International Conference on Electronic Commerce (ICEC'05). ACM Press, New York, 287--292. Google ScholarDigital Library
- R. S. Sutton and A. G. Barto. 1998. Reinforcement Learning: An Introduction. Bradford. Google ScholarDigital Library
- C. Szepesvári. 2010. Algorithms for Reinforcement Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning Series. Vol. 4, Morgan and Claypool, San Rafael, CA. Google ScholarDigital Library
- N. Taghipour and A. Kardan. 2008. A hybrid web recommender system based on q-learning. In Proceedings of the ACM Symposium on Applied Computing (SAC'08). 1164--1168. Google ScholarDigital Library
- X. Wang, D. Rosenblum, and Y. Wang. 2012. Context-aware mobile music recommendation for daily activities. In Proceedings of the 20th ACM International Conference on Multimedia (MM'12). ACM Press, New York, 99--108. Google ScholarDigital Library
- K. Yoshii, M. Goto, K. Komatani, T. Ogata, and H. G. Okuno. 2006. Hybrid collaborative and content-based music recommendation using probabilistic model with latent user preferences. In Proceedings of the 7th International Conference on Music Information Retrieval (ISMIR'06). 296--301.Google Scholar
- B.-T. Zhang and Y.-W. Seo. 2001. Personalized web-document filtering using reinforcement learning. Appl. Artif. Intell. 15, 665--685.Google ScholarCross Ref
- Y. C. Zhang, Diarmuid, D. Quercia, and T. Jambor. 2012. Auralist: Introducing serendipity into music recommendation. In Proceedings of the 5th ACM International Conference on Web Search and Data Mining (WSDM'12). ACM Press, New York, 13--22. Google ScholarDigital Library
- E. Zheleva, J. Guiver, E. M. Rodrigues, and N. M. Frayling. 2010. Statistical models of music-listening sessions in social media. In Proceedings of the 19th International Conference on World Wide Web (WWW'10). ACM Press, New York, 1019--1028. Google ScholarDigital Library
Index Terms
Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach
Recommendations
Serendipitous Personalized Ranking for Top-N Recommendation
WI-IAT '12: Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01Serendipitous recommendation has benefitted both e-retailers and users. It tends to suggest items which are both unexpected and useful to users. These items are not only profitable to the retailers but also surprisingly suitable to consumers' tastes. ...
Effects of Personalized and Aggregate Top-N Recommendation Lists on User Preference Ratings
Prior research has shown a robust effect of personalized product recommendations on user preference judgments for items. Specifically, the display of system-predicted preference ratings as item recommendations has been shown in multiple studies to bias ...
A novel method for personalized music recommendation
With the development of digital music technologies, it is an interesting and useful issue to recommend the 'favored music' from large amounts of digital music. Some Web-based music stores can recommend popular music which has been rated by many people. ...
Comments