Yusuke Yamamoto
Yasuaki Kakehi
ABSTRACT
This paper presents a novel musical performance system named onNote that directly utilizes printed music scores as a musical instrument. This system can make users believe that sound is indeed embedded on the music notes in the scores. The users can play music simply by placing, moving and touching the scores under a desk lamp equipped with a camera and a small projector. By varying the movement, the users can control the playing sound and the tempo of the music. To develop this system, we propose an image processing based framework for retrieving music from a music database by capturing printed music scores. From a captured image, we identify the scores by matching them with the reference music scores, and compute the position and pose of the scores with respect to the camera. By using this framework, we can develop novel types of musical interactions.
Supplemental Material
- http://www.free-scores.com/.Google Scholar
- A. Cassinelli, Y. Kuribara, A. Zerroug, M. Ishikawa, and D.Manabe. scoreLight: playing with a humansized laser pickup. In NIME, pages 144--149, 2010.Google Scholar
- E. Costanza, M. Giaccone, O. Kueng, S. Shelley, and J. Huang. Tangible interfaces for download: initial observations from users' everyday environments. In CHI, pages 2765--2774, 2010. Google ScholarDigital Library
- C. Dalitz, M. Droettboom, B. Pranzas, and I. Fujinaga. A comparative study of staff removal algorithms. PAMI, 30(5):753--766, 2008. Google ScholarDigital Library
- P. L. Davidson and J. Y. Han. Synthesis and control on large scale multi-touch sensing displays. In NIME, pages 216--219, 2006. Google ScholarDigital Library
- W. B. Davis, K. E. Gfeller, and M. H. Thaut. An Introduction to Music Therapy: Theory and Practice. Third Edition. AMTA, third edition, 2008.Google Scholar
- D. Doermann. The indexing and retrieval of document images: a survey. CVIU, 70:287--298, 1998. Google ScholarDigital Library
- M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. C. of the ACM, 24:381--395, 1981. Google ScholarDigital Library
- A. Fornes, J. Llados, G. Sanchez, and H. Bunke. On the use of textural features for writer identification in old handwritten music scores. In ICDAR, pages 996--1000, 2009. Google ScholarDigital Library
- A. Forsberg, M. Dieterich, and R. Zeleznik. The music notepad. In UIST, pages 203--210, 1998. Google ScholarDigital Library
- C. Fremerey, M. Muller, and F. K. M. Clausen. Automatic mapping of scanned sheet music to audio recordings. In ISMIR, 2008.Google Scholar
- I. Fujinaga, B. Alphonce, B. Pennycook, and K. Hogan. Optical music recognition: Progress report. In ICMC, pages 66--73, 1991.Google Scholar
- M. Goto. Smart Music KIOSK: music listening station with chorus-search function. In UIST, pages 31--40, 2003. Google ScholarDigital Library
- S. Jorda and M. Alonso. Mary had a little scoretable* or the reactable* goes melodic. In NIME, pages 208--211, 2006. Google ScholarDigital Library
- S. Jorda, G. Geiger, M. Alonso, and M. Kaltenbrunner. The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. In TEI, pages 139--146, 2007. Google ScholarDigital Library
- H. Ketabdar, A. Jahanbekam, K. A. Yuksel, T. Hirsch, and A. Haji Abolhassani. MagiMusic: using embedded compass (magnetic) sensor for touch-less gesture based interaction with digital music instruments in mobile devices. In TEI, pages 241--244, 2011. Google ScholarDigital Library
- D. G. Lowe. Distinctive image features from scaleinvariant keypoints. IJCV, 60:91--110, 2004. Google ScholarDigital Library
- H. Miyao and M. Maruyama. An online handwritten music symbol recognition system. IJDAR, 9(1):49--58, 2007. Google ScholarDigital Library
- T. Nakai, K. Kise, and M. Iwamura. Use of affine invariants in locally likely arrangement hashing for camera-based document image retrieval. In DAS, pages 541--552, 2006. Google ScholarDigital Library
- Y. Nishibori and T. Iwai. TENORI-ON. In NIME, pages 172--175, 2006. Google ScholarDigital Library
- D. Nister and H. Stewenius. Scalable recognition with a vocabulary tree. In CVPR, pages 2161--2168, 2006. Google ScholarDigital Library
- D. Overholt. Visually controlled synthesis using the sonic scanner and the graphonic interface. In AES Convention, 2004.Google Scholar
- D. Overholt, J. Thompson, L. Putnam, B. Bell, J. Kleban, B. Sturm, and J. Kuchera-Morin. A multimodal system for gesture recognition in interactive music performance. CMJ, 33:69--82, 2009. Google ScholarDigital Library
- J. Patten, B. Recht, and H. Ishii. Interaction techniques for musical performance with tabletop tangible interfaces. In ACE, 2006. Google ScholarDigital Library
- J. C. Pinto, P. Vieira, and J. M. Sousa. A new graphlike classification method applied to ancient handwritten musical symbols. IJDAR, 6(1):10--22, 2004.Google ScholarDigital Library
- A. Rebelo, G. Capela, and J. S. Cardoso. Optical recognition of music symbols: A comparative study. IJDAR, 13:19--31, 2010. Google ScholarDigital Library
- H. Uchiyama and H. Saito. Augmenting text document by on-line learning of local arrangement of keypoints. In ISMAR, pages 95--98, 2009. Google ScholarDigital Library
- E. L. Wong, W. Y. F. Yuen, and C. S. T. Choy. Designing wii controller: a powerful musical instrument in an interactive music performance system. In MoMM, pages 82--87, 2008. Google ScholarDigital Library
Index Terms
- onNote: playing printed music scores as a musical instrument
Recommendations
onNote: a musical interface using markerless physical scores
SIGGRAPH '11: ACM SIGGRAPH 2011 PostersIntuitive music playing using various digital musical instruments with specific tangible interfaces has become one of the ways to enjoy the music experience [Jorda et al. 2007]. When we start studying music, we need to learn how to read musical scores ...
Guitar man: (an implementation of a rhythm game cooperative musical performance system with actual musical instruments)
SIGGRAPH Asia '08: ACM SIGGRAPH ASIA 2008 educators programmeThis study proposes a method for learning musical instruments using computer graphics. The conventional musical instrument learning systems have been considered as simple game controllers or simple score processors. This study produced an education ...
ism: improvisation supporting system based on melody correction
NIME '04: Proceedings of the 2004 conference on New interfaces for musical expressionIn this paper, we describe a novel improvisation supporting system based on correcting musically unnatural melodies. Since improvisation is the musical performance style that involves creating melodies while playing, it is not easy even for the people ...
Comments