MirrorFugue: communicating hand gesture in remote piano collaboration

ABSTRACT

Playing a musical instrument involves a complex set of continuous gestures, both to play the notes and to convey expression. To learn an instrument, a student must learn not only the music itself but also how to perform these bodily gestures. We present MirrorFugue, a set of three interfaces on a piano keyboard designed to visualize hand gesture of a remote collaborator. Based their spatial configurations, we call our interfaces Shadow, Reflection, and Organ. We describe the configurations and detail studies of our designs on synchronous, remote collaboration, focusing specifically on remote lessons for beginners. Based on our evaluations, we conclude that displaying the to-scale hand gestures of a teacher at the locus of interaction can improve remote piano learning for novices.

References

1. Barbosa A. 2003. Displaced soundscapes: a survey of network systems for music and sonic-art creation. Leonardo Music Journal, Volume 13, Issue 1, 53--59.
2. Burk, P. L. 2000. Jammin' on the web: a new client/server architecture for multi-user performance. Proceedings of the 2000 International Computer Music Conference. San Francisco, California: International Computer Music Association, 117--120.
3. Casio. http://www.casio.com.
4. Gang, D., Chockler, G. V., Anker, T., and Kremer, A. 1997. TransMIDI: a system for midi sessions over the network using transis. In Proceedings of the 1999 International Computer Music Conference. San Francisco, California: International Computer Music Association, 283--286.
5. Ishii, H. and Kobayashi, M. 1992. ClearBoard: a seamless medium for shared drawing and conversation with eye contact. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey CA, May 1992). P. Bauersfeld, J. Bennett, and G. Lynch, Eds. CHI '92. ACM, New York, NY, 525--532.
6. Ishii, H. 1990. TeamWorkStation: towards a seamless shared workspace. In Proceedings of the 1990 ACM Conference on Computer-Supported Cooperative Work (Los Angeles CA, October 1990). CSCW '90. ACM, New York, NY, 13--26.
7. Klemmer, S. R., Hartmann, B., and Takayama, L. 2006. How bodies matter: five themes for interaction design. In Proceedings of the 6th Conference on Designing interactive Systems (University Park PA, June 2006). DIS '06. ACM, New York, NY, 140--149.
8. Konstantas, D. 1997. Distributed musical rehearsal. In Proceedings of the 1997 International Computer Music Conference. San Francisco, California: International Computer Music Association, 279--282.
9. Krueger, M. W., Gionfriddo, T., and Hinrichsen, K. 1985. VIDEOPLACE--an artificial reality. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Francisco, CA). CHI '85. ACM, New York, NY, 35--40.
10. Miwa, Y. and Ishibiki, C. 2004. Shadow communication: system for embodied interaction with remote partners. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (Chicago IL, November 2004). CSCW '04. ACM, New York, NY, 467--476.
11. Moog Piano Bar. http://www.moogmusic.com/.
12. Morikawa, O. and Maesako, T. 1997. HyperMirror: a video-mediated communication system. In CHI '97 Extended Abstracts on Human Factors in Computing Systems: Looking To the Future (Atlanta GA, March 1997). CHI '97. ACM, New York, NY, 317--318.
13. MusicPath: Networking People and Music. http:// musicpath.acadiau.ca/.
14. Ng, K. 2008. Interactive feedbacks with visualisation and sonification for technology-enhanced learning for music performance. In Proceedings of the 26th Annual ACM international Conference on Design of Communication (Lisbon, Portugal, September 22-24, 2008). SIGDOC '08. ACM, New York, NY, 281--282.
15. Tang, J. C. and Minneman, S. L. 1990. VideoDraw: a video interface for collaborative drawing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering People (Seattle WA, April 1990). J. C. Chew and J. Whiteside, Eds. CHI '90. ACM, New York, NY, 313--320.
16. Tang, J. C. and Minneman, S. 1991. VideoWhiteboard: video shadows to support remote collaboration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Reaching Through Technology (New Orleans LA, April 1991). S. P. Robertson, G. M. Olson, and J. S. Olson, Eds. CHI '91. ACM, New York, NY, 315--322.
17. Wanderly, M. 2001. Performer-instrument interaction: applications to gestural control of sound synthesis. PhD Thesis, University Paris VI.
18. Wellner, P. and Newman, W. 1992. A desk supporting computer-based interaction with paper documents. In Proceedings CHI 92 Conference, 587--592.
19. Yamaha. http://music.yamaha.com/.
20. Young, J. and Fujinaga, I. 1999. Piano master classes via the Internet. In Proceedings of the 1999 International Computer Music Conference. San Francisco, California: International Computer Music Association, pp. 135--137.
21. Zimmermann, R., Chew, E., Ay, S. A., and Pawar, M. 2008. Distributed musical performances: Architecture and stream management. ACM Trans. Multimedia Comput. Commun. Appl. 4, 2 (May. 2008), 1--23..

Index Terms

• Human-centered computing

  • Collaborative and social computing

    • Collaborative and social computing theory, concepts and paradigms

      • Computer supported cooperative work

• Social and professional topics

  • Professional topics

    • Computing and business

      • Computer supported cooperative work