Distributed musical performances: Architecture and stream management

ABSTRACT

An increasing number of novel applications produce a rich set of different data types that need to be managed efficiently and coherently. In this article we present our experience with designing and implementing a data management infrastructure for a distributed immersive performance (DIP) application. The DIP project investigates a versatile framework for the capture, recording, and replay of video, audio, and MIDI (Musical Instrument Digital Interface) streams in an interactive environment for collaborative music performance. We are focusing on two classes of data streams that are generated within this environment. The first category consists of high-resolution isochronous media streams, namely audio and video. The second class comprises MIDI data produced by electronic instruments. MIDI event sequences are alphanumeric in nature and fall into the category of the data streams that have been of interest to data management researchers in recent years.

We present our data management architecture, which provides a repository for all DIP data. Streams of both categories need to be acquired, transmitted, stored, and replayed in real time. Data items are correlated across different streams with temporal indices. The audio and video streams are managed in our own High-performance Data Recording Architecture (HYDRA), which integrates multistream recording and retrieval in a consistent manner. This paper reports on the practical issues and challenges that we encountered during the design, implementation and experimental phases of our prototype. We also present some analysis results and discuss future extensions for the architecture.

REFERENCES

• 1: Blakowski, G. and Steinmetz, R. 1996. A media synchronization survey: Reference model, specification, and case studies. IEEE J. Select. Areas Comm. 14, 1, 5--35.
• 2: Bolosky, W. J., Barrera, J. S., Draves, R. P., Fitzgerald, R. P., Gibson, G. A., Jones, M. B., Levi, S. P., Myhrvold, N. P., and Rashid, R. F. 1996. The tiger video fileserver. In Proceedings of the 6th Workshop on Network and Operating System Support for Digital Audio and Video. Zushi, Japan.
• 3: CCRMA. 2002. Stanford University's SoundWire Group at CCRMA, jam session. www.ccrma.stanford.edu/groups/soundwire.
• 4: Chafe, C. 2005. Low-latency audio, networked musical performance experiments. ccrma.stanford.edu/~cc.
• 5: Chafe, C., Gurevich, M., Leslie, G., and Tyan, S. 2004. Effect of time delay on ensemble accuracy. In Proceedings of the International Symposium on Musical Acoustics (ISMA'04). Nara, Japan.
• 6: Chafe, C., Wilson, S., Leistikow, R., Chisholm, D., and Scavone, G. 2000. A simplified approach to high quality music and sound over IP. In Proceedings of the COST G-6 Conference on Digital Audio Effects (DAFX'00). Verona, Italy.
• 7: Chandrasekaran, S. 2003. Telegraphcq: Continuous dataflow processing for an uncertain world. In Proceedings of the 1st Biennial Conference on Innovative Data Systems Research (CIDR 2003). Asilomar, CA.
• 8: Chew, E., Sawchuk, A., Tanoue, C., and Zimmermann, R. 2005. Segmental tempo analysis of performances in performer-centered experiments in the distributed immersive performance project. In Proceedings of International Conference on Sound and Music Computing (SMC'05). Salerno, Italy.
• 9: Chew, E., Sawchuk, A., Zimmermann, R., the Tosheff Piano Duo (Stoyanova, V., Tosheff, I.), Kyriakakis, C., Papadopoulos, C., François, A., and Volk, A. 2004. Distributed immersive performance. In Proceedings of the Annual National Association of the Schools of Music Meeting (NASM'04). San Diego, CA.
• 10: Chew, E., Zimmermann, R., Sawchuk, A., Kyriakakis, C., Papadopoulos, C., François, A., Kim, G., and Volk, A. 2004. Musical interaction at a distance: Distributed immersive performance. In Proceedings of the Open Workshop of MUSICNETWORK: Integration of Music in Multimedia Applications (MUSICNETWORK'04). Barcelona, Spain.
• 11: Chew, E., Zimmermann, R., Sawchuk, A., Papadopoulos, C., Kyriakakis, C., Tanoue, C., Desai, D., Pawar, M., Sinha, R., and Meyer, W. 2005. A second report on the user experiments in the distributed immersive performance project. In Proceedings of the Open Workshop of MUSICNETWORK: Integration of Music in Multimedia Applications (MUSICNETWORK'05). Vienna, Austria.
• 12: Cooperstock, J. R. and Spackman, S. P. 2001. The recording studio that spanned a continent. In Proceedings of the IEEE International Conference on Web Delivering of Music (WEDELMUSIC'01). Florence, Italy.
• 13: Ghandeharizadeh, S., Zimmermann, R., Shi, W., Rejaie, R., Ierardi, D., and Li, T. 1997. Mitra: A scalable continuous media server. Kluwer Multimedia Tools Appl. 5, 1, 79--108.
• 14: Gresham-Lancaster, S. 2005. AB_Time Vancouver-Marseilles-Troy concert at NIME'05. Vancouver, Canada. hct.ece.ubc.ca/nime/2005/concerts.html.
• 15: Gu, X., Dick, M., Kurtisi, Z., Noyer, U., and Wolf, L. 2005. Network-centric music performance: Practice and experiments. IEEE Comm. Mag. 43, 6, 86--93.
• 16: Hsieh, J., Liu, J., Du, D., Ruwart, T., and Lin, M. 1995. Experimental performance of a mass storage system for video-on-demand. (Special Issue of Multimedia Systems and Technology) J. Parall. Distrib. Comput. (JPDC) 30, 2, 147--167.
• 17: Internet2 Member Meeting. 2004. The University of Texas at Austin and Internet2 to Host first virtual concert experiment. Internet2 Member Meeting, Austin, TX. www.utexas.edu/opa/news/04newsreleases/nr_200409/nr_internet2_040927.html#info.
• 18: Kanki, S. 1998. Mélange à trois. NOTAM. www.notam02.no/warsaw/melange.html.
• 19: Konstantas, D. 1998. Overview of telepresence environment for distributed musical rehearsals. In Proceedings of the ACM Symposium on Applied Computing (SAC'98). Atlanta, GA.
• 20: Lambrinos, L., Kirstein, P., and Hardman, V. 1998. The multicast multimedia conference recorder. In Proceedings of the International Conference on Computer Communications and Networks (IC3N). Lafayette, LA.
• 21: Laursen, A., Olkin, J., and Porter, M. 1994. Oracle media server: Providing consumer based interactive access to multimedia data. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 470--477.
• 22: Lazzaro, J. and Wawrzynek, J. 2001. A case for network musical performance. In Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'01). New York, NY.
• 23: Lie, A. and Rønningen, L. A. 2003. Distributed multimedia plays with QoS guaranties over IP. In Proceedings of the International Conference on WEB Delivering of Music (WEDELMUSIC'03). Leeds, UK.
• 24: Martin, C., Narayan, P. S., Özden, B., Rastogi, R., and Silberschatz, A. 1996. The fellini multimedia storage server. In Multimedia Information Storage and Management, S. M. Chung, Ed. Kluwer Academic Publishers, Boston, MA, Chapter 5.
• 25: McLeod, D., Neumann, U., Nikias, C., and Sawchuk, A. 1999. Integrated media systems. IEEE Signal Process. Magazine 16, 33--76.
• 26: Mills, D. L. 1991. Internet time synchronization: The network time protocol. IEEE Trans. Comm. 38, 10, 1482--1493.
• 27: Muntz, R., Santos, J., and Berson, S. 1997. RIO: A real-time multimedia object server. ACM Sigmet. Perform. Eval. Rev. 25, 2, 29--35.
• 28: Perrot, X. 2004. 2004 Berlin-Paris network concert at ICHIM'04. Berlin, Germany. www.ichim.org/jahia/Jahia/lang/en.
• 29: PostgreSQL Group. 2003. PostgreSQL Open Source database system. The PostgreSQL Group. www.postgresql.org.
• 30: Robbins, S. 1995. A MIDI primer for computer scientists. Division of Computer Science Tech. Rep. CS-95-11, University of Texas, San Antonio.
• 31: Rosen, T. 2004. Is it live or is it Internet? Austin, Texas. www.utexas.edu/features/archive/2004/internet.html.
• 32: Sawchuk, A. A., Chew, E., Zimmermann, R., Papadopoulos, C., and Kyriakakis, C. 2003. From remote media immersion to distributed immersive performance. In Proceedings of the ACM SIGMM 2003 Workshop on Experiential Telepresence (ETP 2003). Berkeley, CA.
• 33: Schooler, E. 2001. Distributed music: A foray into networked performance, Haydn piano Trio No.1 in G, the finale movement. www.postel.org/pipermail/end2end-interest/2001-August/001314.html.
• 34: Schulzrinne, H., Casner, S., Frederick, R., and Jacobson, V. 1996. RTP: A transport protocol for real time applications. www.ietf.org/rfc/rfc1889.txt.
• 35: Schulzrinne, H., Rao, A., and Lanphier, R. 1998. Real time streaming protocol (RTSP). www.ietf.org/rfc/rfc2326.txt.
• 36: Shahabi, C., Zimmermann, R., Fu, K., and Yao, S.-Y. D. 2002. Yima: A second generation continuous media server. IEEE Comput. 35, 6, 56--64.
• 37: Tobagi, F., Pang, J., Baird, R., and Gang, M. 1993. Streaming RAID-A disk array management system for video files. In Proceedings of the 1st ACM International Conference on Multimedia. Anaheim, CA, 393--400.
• 38: Witherspoon, J. T., Schuckman, L., and Spilker, J. 1978. Time transfer via GPS. In Proceedings of the Annual Precise Time and Time Interval Applications and Planning Meeting. Goddard Space Flight Center, Greenbelt, MD.
• 39: Xu, A., Woszczyk, W., Settel, Z., Pennycook, B., Rowe, R., Galanter, P., Bary, J., Martin, G., Corey, J., and Cooperstock, J. R. 2000. Real-time streaming of multichannel audio data over Internet. J. Audio Eng. Soc. 48, 7--8.
• 40: Young, J. P. and Fujinaga, I. 1999. Piano master classes via the Internet. In Proceedings of the International Computer Music Conference (ICMA'99). Beijing, China.
• 41: Zimmermann, R., Fu, K., Jahangiri, M., and Shahabi, C. 2006. A multi-threshold online smoothing technique for variable rate multimedia streams. Multimedia Tools Appl. 28, 1.
• 42: Zimmermann, R., Fu, K., and Ku, W.-S. 2003. Design of a large scale data stream recorder. In Proceedings of the 5th International Conference on Enterprise Information Systems (ICEIS'03). Angers, France. www.iceis.org.
• 43: Zimmermann, R., Fu, K., Shahabi, C., Yao, S.-Y. D., and Zhu, H. 2001. Yima: Design and evaluation of a streaming media system for residential broadband services. In Proceedings of the VLDB Workshop on Databases in Telecommunications (DBTel'01). Rome, Italy.
• 44: Zimmermann, R., Kyriakakis, C., Shahabi, C., Papadopoulos, C., Sawchuk, A. A., and Neumann, U. 2004. RMI system: Internet meets the future home theater. IEEE Multimedia (Special issue on “Digital Multimedia on Demand”) 11, 2, 48--57.
• 45: Zimmermann, R., Shahabi, C., Fu, K., and Yao, S.-Y. D. 2004. Scalability evaluation of the Yima streaming media architecture. Softw. Pract. Exper. 35, 4, 345--359.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.2 Information Storage

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Multimedia databases

  J. Computer Applications
  J.5 ARTS AND HUMANITIES

General Terms:
Design, Experimentation, Measurement

Keywords:
Distributed immersive performance, multimedia storage, multimodal data recorder, networked musical performance