Towards musical query-by-semantic-description using the CAL500 data set

ABSTRACT

Query-by-semantic-description (QBSD)is a natural paradigm for retrieving content from large databases of music. A major impediment to the development of good QBSD systems for music information retrieval has been the lack of a cleanly-labeled, publicly-available, heterogeneous data set of songs and associated annotations. We have collected the Computer Audition Lab 500-song (CAL500) data set by having humans listen to and annotate songs using a survey designed to capture 'semantic associations' between music and words. We adapt the supervised multi-class labeling (SML) model, which has shown good performance on the task of image retrieval, and use the CAL500 data to learn a model for music retrieval. The model parameters are estimated using the weighted mixture hierarchies expectation-maximization algorithm which has been specifically designed to handle real-valued semantic association between words and songs, rather than binary class labels. The output of the SML model, a vector of class-conditional probabilities, can be interpreted as a semantic multinomial distribution over a vocabulary. By also representing a semantic query as a query multinomial distribution, we can quickly rank order the songs in a database based on the Kullback-Leibler divergence between the query multinomial and each song's semantic multinomial. Qualitative and quantitative results demonstrate that our SML model can both annotate a novel song with meaningful words and retrieve relevant songs given a multi-word, text-based query.

REFERENCES

• 1: Gustavo Carneiro , Antoni B. Chan , Pedro J. Moreno , Nuno Vasconcelos, Supervised Learning of Semantic Classes for Image Annotation and Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.29 n.3, p.394-410, March 2007 [doi>10.1109/TPAMI.2007.61]
• 2: S.L. Feng, R. Manmatha, and Victor Lavrenko. Multiple bernoulli relevance models for image and video annotation. IEEE CVPR 2004.
• 3: David M. Blei , Michael I. Jordan, Modeling annotated data, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860460]
• 4: D. A. Forsyth , M. M. Fleck, Body plans, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.678, June 17-19, 1997
• 5: International conferences of music information retrieval. http://www.ismir.net/.
• 6: MIREX 2005. Music information retrieval evaluation exchange. http://www.music-ir.org/mirex 2005.
• 7: M. Goto and K. Hirata. Recent studies on music information processing. Acoustical Science and Technology 25(4):419--425, 2004.
• 8: R.B. Dannenberg and N. Hu. Understanding search performance in query-by-humming systems. ISMIR 2004.
• 9: A. Kapur, M. Benning, and G. Tzanetakis. Query by beatboxing: Music information retrieval for the dj. ISMIR 2004.
• 10: G. Eisenberg, J.M. Batke, and T. Sikora. Beatbank - an mpeg-7 compliant query by tapping system. Audio Engineering Society Convention 2004.
• 11: M.F. McKinney and J. Breebaart. Features for audio and music classification. ISMIR 2003.
• 12: T. Li and G. Tzanetakis. Factors in automatic musical genre classification of audio signals. IEEE WASPAA 2003.
• 13: S. Essid, G. Richard, and B. David. Inferring efficient hierarchical taxonomies for music information retrieval tasks: Application to musical instruments. ISMIR 2005.
• 14: F. Pachet and D. Cazaly. A taxonomy of musical genres. RIAO 2000.
• 15: Adam Berenzweig , Beth Logan , Daniel P. W. Ellis , Brian P. W. Whitman, A Large-Scale Evaluation of Acoustic and Subjective Music-Similarity Measures, Computer Music Journal, v.28 n.2, p.63-76, June 2004 [doi>10.1162/014892604323112257]
• 16: Brian A. Whitman , Barry L. Vercoe, Learning the meaning of music, Massachusetts Institute of Technology, Cambridge, MA, 2005
• 17: B. Whitman and D. Ellis. Automatic record reviews. ISMIR 2004.
• 18: B. Whitman and R. Rifkin. Musical query-by-description as a multiclass learning problem. IEEE Workshop on Multimedia Signal Processing 2002.
• 19: M. Slaney. Semantic-audio retrieval. IEEE ICASSP 2002.
• 20: P. Cano and M. Koppenberger. Automatic sound annotation. In IEEE workshop on Machine Learning for Signal Processing 2004.
• 21: M. Slaney. Mixtures of probability experts for audio retrieval and indexing. IEEE Multimedia and Expo 2002.
• 22: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 23: N. Vasconcelos. Image indexing with mixture hierarchies. IEEE CVPR pages 3--10, 2001.
• 24: D. Turnbull, L. Barrington, and G. Lanckriet. Modelling music and words using a multi-class naíve bayes approach. ISMIR 2006.
• 25: C. McKay, D. McEnnis, and I. Fujinaga. A large publicly accessible prototype audio database for music research. ISMIR 2006.
• 26: J. Skowronek, M. McKinney, and S. ven de Par. Ground-truth for automatic music mood classification. ISMIR 2006.
• 27: Xiao Hu, J.S. Downie, and A.F. Ehmann. Exploiting recommended usage metadata: Exploratory analyses. ISMIR 2006.
• 28: Lawrence Rabiner , Biing-Hwang Juang, Fundamentals of speech recognition, Prentice-Hall, Inc., Upper Saddle River, NJ, 1993
• 29: L. Barrington, A. Chan, D. Turnbull, and G. Lanckriet. Audio information retrieval using semantic similarity. In IEEE ICASSP 2007.
• 30: D. Turnbull, R. Liu, L. Barrington, D. Torres,and G. Lanckriet. Using games to collect semantic information about music. Technical report, 2007.
• 31: Luis von Ahn, Games with a Purpose, Computer, v.39 n.6, p.92-94, June 2006 [doi>10.1109/MC.2006.196]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.m Miscellaneous

Additional Classification:
  J. Computer Applications
  J.5 ARTS AND HUMANITIES

General Terms:
Algorithms, Design, Experimentation

Keywords:
content-based music information retrieval, query-by-semantic-description, supervised multi-class classification