skip to main content
10.1145/3025453.3025900acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Playing Fast and Loose with Music Recognition

Published: 02 May 2017 Publication History

Abstract

We report lessons from iteratively developing a music recognition system to enable a wide range of musicians to embed musical codes into their typical performance practice. The musician composes fragments of music that can be played back with varying levels of embellishment, disguise and looseness to trigger digital interactions. We collaborated with twenty-three musicians, spanning professionals to amateurs and working with a variety of instruments. We chart the rapid evolution of the system to meet their needs as they strove to integrate music recognition technology into their performance practice, introducing multiple features to enable them to trade-off reliability with musical expression. Collectively, these support the idea of deliberately introducing "looseness" into interactive systems by addressing the three key challenges of control, feedback and attunement, and highlight the potential role for written notations in other recognition-based systems.

Supplementary Material

suppl.mov (pn3447-file3.mp4)
Supplemental video
suppl.mov (pn3447p.mp4)
Supplemental video
MP4 File (p4302-greenhalgh.mp4)

References

[1]
Michel Beaudouin-Lafon and Wendy Mackay. 2002. Prototyping tools and techniques. In The humancomputer interaction handbook, Julie A. Jacko and Andrew Sears (Eds.). L. Erlbaum Associates Inc., Hillsdale, NJ, USA 1006--1031.
[2]
Victoria Bellotti, Maribeth Back, W. Keith Edwards, Rebecca E. Grinter, Austin Henderson, and Cristina Lopes. 2002. Making sense of sensing systems: five questions for designers and researchers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, NY, NY, USA, 415--422. DOI=http://dx.doi.org/10.1145/503376.503450
[3]
Emmanouil Benetos and Simon Dixon. 2012. A ShiftInvariant Latent Variable Model for Automatic Music Transcription. Comput. Music J. 36, 4 (Dec.2012), 81--94.
[4]
Steve Benford, Holger Schnädelbach, Boriana Koleva, Rob Anastasi, Chris Greenhalgh, Tom Rodden, Jonathan Green, Ahmed Ghali, Tony Pridmore, Bill Gaver, Andy Boucher, Brendan Walker, Sarah Pennington, Albrecht Schmidt, Hans Gellersen, and Anthony Steed. 2005. Expected, sensed, and desired: A framework for designing sensing-based interaction. ACM Trans. Comput.-Hum. Interact. 12, 1 (March 2005), 3--30. DOI=http://dx.doi.org/10.1145/1057237.1057239
[5]
Mason Bretan and Gil Weinberg. 2016. A survey of robotic musicianship. Commun. ACM 59, 5 (April 2016), 100--109.
[6]
Chris Cannam, Matthias Mauch, Matthew E. P. Davies, Simon Dixon, Christian Landone, Katy C. Noland, Mark Levy, Massimiliano Zanoni, Dan Stowell, and Luís A. Figueira, 2013. MIREX 2013 entry: Vamp plugins from the centre for digital music. MIREX.
[7]
Alan J. Dix. 1992. Pace and interaction. In Proceedings of HCI'92: People and Computers VII, Eds. A. Monk, D. Diaper and M. Harrison. Cambridge University Press. 193--207.
[8]
Alan J. Dix. 1995. Cooperation without (reliable) Communication: Interfaces for Mobile Applications. Distributed Systems Engineering, 2(3): pp. 171--181.
[9]
Tom Djajadiningrat, Kees Overbeeke, and Stephan Wensveen. 2002. But how, Donald, tell us how?: on the creation of meaning in interaction design through feedforward and inherent feedback. In Proceedings of the 4th conference on Designing interactive systems: processes, practices, methods, and techniques (DIS '02). ACM, NY, NY, USA, 285--291. DOI=http://dx.doi.org/10.1145/778712.778752
[10]
Kenneth H. Goodrich, Paul C. Schutte, Frank O. Flemisch and Ralph A. Williams. 2006. Application of the H-Mode, A Design and Interaction Concept for Highly Automated Vehicles, to Aircraft. 2006 ieee/aiaa 25TH Digital Avionics Systems Conference, Portland, OR, 1--13.
[11]
Chris Greenhalgh, Steve Benford, Adrian Hazzard. 2016. ^muzicode$: Composing and Performing Musical Codes. In Proceedings of AM '16, October 04 -- 06. Norrköping, Sweden. http://dx.doi.org/10.1145/2986416.2986444
[12]
Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B. Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, Nicolas Roussel, and Björn Eiderbäck. 2003. Technology probes: inspiring design for and with families. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, NY, NY, USA, 17--24. DOI=http://dx.doi.org/10.1145/642611.642616
[13]
ISO/IEC. 1999. 13407 Human-Centred Design Processes for Interactive Systems, ISO/IEC 13407: 1999 (E).
[14]
C. Joder, S. Essid, and G. Richard. 2011. A Conditional Random Field Framework for Robust and Scalable Audio-to-Score Matching. Trans. Audio, Speech and Lang. Proc. 19, 8 (November 2011), 23852397. DOI=http://dx.doi.org/10.1109/TASL.2011.2134092
[15]
Anna Jordanous and Alan Smaill. 2009. Investigating the Role of Score Following in Automatic Musical Accompaniment. Journal of New Music Research, 38 (2). 197 - 209.
[16]
Olivier Lartillot and Petri Toiviainen. 2007. A Matlab toolbox for musical feature extraction from audio, In International Conference on Digital Audio Effects, 237--244.
[17]
Tod Machover and Joe Chung. 1989. Hyperinstruments: Musically Intelligent and Interactive Performance and Creativity Systems. In ICMC 1989, 186--190.
[18]
Robert Cecil Martin. 2003. Agile Software Development: Principles, Patterns, and Practices. Prentice Hall PTR, Upper Saddle River, NJ, USA.
[19]
Andrew McPherson. 2010. The Magnetic Resonator Piano: Electronic Augmentation of an Acoustic Grand Piano. Journal of New Music Research, 39(3), 189202. DOI=10.1080/09298211003695587
[20]
Donald A. Norman and Stephen W. Draper. 1986. User Centered System Design; New Perspectives on Human-Computer Interaction. L. Erlbaum Assoc. Inc., Hillsdale, NJ, USA.
[21]
Donald A. Norman. 2010. Natural user interfaces are not natural. interactions 17, 3 (May 2010), 6--10. DOI=http://dx.doi.org/10.1145/1744161.1744163
[22]
Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, NY, NY, USA, 147--156. DOI=http://dx.doi.org/10.1145/1978942.1978965.
[23]
François Pachet. 2004. On the design of a musical flow machine. In A Learning Zone of One's Own, Tokoro and Steels (Eds), IOS Press, 111--134.
[24]
George Perle. 1972. Serial composition and atonality: an introduction to the music of Schoenberg, Berg, and Webern. Univ of California Press.
[25]
Henning Pohl and Roderick Murray-Smith. 2013. Focused and casual interactions: allowing users to vary their level of engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 22232232.
[26]
R. Rowe. 1992. Machine Listening and Composing with Cypher. Comput. Music J. 16, 1, 43.
[27]
Justin Salamon, Emilia Gomez, Daniel P. W. Ellis, and Gael Richard. 2014. Melody extraction from polyphonic music signals: Approaches, applications, and challenges. Signal Process. Mag. IEEE 31, 2--134.
[28]
E. Sams. 1970. Elgar's Cipher Letter to Dorabella. Music. Times, 151--154.
[29]
A. M. Stark. 2014. Sound Analyser: A Plug-In For Real-Time Audio Analysis In Live Performances And Installations. In Proceedings of New Interfaces for Musical Expression (NIME), London, 2014
[30]
Greg Turner, Alastair Weakley, Yun Zhang, Ernest Edmonds. 2005. Attuning: A Social and Technical Study of Artist--Programmer Collaborations. In 17th Workshop of the Psychology of Programming Interest Group, Sussex University, P. Romero, J. Good, E. Acosta Chaparro & S. Bryant (Eds). Proc. PPIG 17, 106--109, June 2005
[31]
Jo Vermeulen, Kris Luyten, Elise van den Hoven, and Karin Coninx. 2013. Crossing the bridge over Norman's Gulf of Execution: revealing feedforward's true identity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 19311940.
[32]
Avery Wang. 2006. The Shazam music recognition service. Commun. ACM 49, 8 (August 2006), 44--48. DOI=http://dx.doi.org/10.1145/1145287.1145312
[33]
Bruno Zamborlin, Frederic Bevilacqua, Marco Gillies, and Mark D'inverno. 2014. Fluid gesture interaction design: Applications of continuous recognition for the design of modern gestural interfaces. ACM Trans. Interact. Intell. Syst. 3, 4, Article 22 (January 2014), 30 pages. DOI=http://dx.doi.org/10.1145/2543921.

Cited By

View all
  • (2024)Jess+: AI and robotics with inclusive music-makingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642548(1-17)Online publication date: 11-May-2024
  • (2024)Augmenting musical instruments with digital identitiesJournal of New Music Research10.1080/09298215.2024.2423613(1-20)Online publication date: 14-Nov-2024
  • (2023)Engaging Passers-by with Rhythm: Applying Feedforward Learning to a Xylophonic Media Architecture FacadeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580761(1-21)Online publication date: 19-Apr-2023
  • Show More Cited By

Index Terms

  1. Playing Fast and Loose with Music Recognition

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. H-metaphor
    2. attunement
    3. casual interactions
    4. looseness
    5. music recognition
    6. notation
    7. performance
    8. sensing systems

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)25
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Jess+: AI and robotics with inclusive music-makingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642548(1-17)Online publication date: 11-May-2024
    • (2024)Augmenting musical instruments with digital identitiesJournal of New Music Research10.1080/09298215.2024.2423613(1-20)Online publication date: 14-Nov-2024
    • (2023)Engaging Passers-by with Rhythm: Applying Feedforward Learning to a Xylophonic Media Architecture FacadeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580761(1-21)Online publication date: 19-Apr-2023
    • (2023)The Stretchy Strap: supporting encumbered interaction with guitarsJournal of New Music Research10.1080/09298215.2023.227483252:1(19-40)Online publication date: 10-Nov-2023
    • (2020)Following the journey of scores through a complex musical workProceedings of the 15th International Audio Mostly Conference10.1145/3411109.3411116(249-252)Online publication date: 15-Sep-2020
    • (2020)Contesting control: journeys through surrender, self-awareness and looseness of control in embodied interactionHuman–Computer Interaction10.1080/07370024.2020.175421436:5-6(361-389)Online publication date: 14-Oct-2020
    • (2019)Encumbered InteractionProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300706(1-13)Online publication date: 2-May-2019
    • (2019)Failing with StyleProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300260(1-14)Online publication date: 2-May-2019
    • (2018)An AI-Based Design Framework to Support Musicians' PracticesProceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion10.1145/3243274.3275381(1-5)Online publication date: 12-Sep-2018
    • (2018)Re-Thinking Immersive Technologies for Audiences of the FutureProceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion10.1145/3243274.3275379(1-3)Online publication date: 12-Sep-2018
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media