Machine Learning Education for Artists, Musicians, and Other Creative Practitioners

Abstract

This article aims to lay a foundation for the research and practice of machine learning education for creative practitioners. It begins by arguing that it is important to teach machine learning to creative practitioners and to conduct research about this teaching, drawing on related work in creative machine learning, creative computing education, and machine learning education. It then draws on research about design processes in engineering and creative practice to motivate a set of learning objectives for students who wish to design new creative artifacts with machine learning. The article then draws on education research and knowledge of creative computing practices to propose a set of teaching strategies that can be used to support creative computing students in achieving these objectives. Explanations of these strategies are accompanied by concrete descriptions of how they have been employed to develop new lectures and activities, and to design new experiential learning and scaffolding technologies, for teaching some of the first courses in the world focused on teaching machine learning to creative practitioners. The article subsequently draws on data collected from these courses—an online course as well as undergraduate and masters-level courses taught at a university—to begin to understand how this curriculum supported student learning, to understand learners’ challenges and mistakes, and to inform future teaching and research.

References

1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16), Vol. 16. 265--283.
2. Charles Ames. 1989. The Markov process as a compositional model: A survey and tutorial. Leonardo 22, 2 (1989), 175--187.
3. Mordechai Ben-Ari. 1998. Constructivism in computer science education. In ACM SIGCSE Bulletin, Vol. 30. ACM, 257--261.
4. Francisco Bernardo, Mick Grierson, and Rebecca Fiebrink. 2018. User-centred design actions for lightweight evaluation of an interactive machine learning toolkit. J. Sci. Technol. Arts 10, 2 (2018), 2--25.
5. Frédéric Bevilacqua, Bruno Zamborlin, Anthony Sypniewski, Norbert Schnell, Fabrice Guédy, and Nicolas Rasamimanana. 2010. Continuous realtime gesture following and recognition. In Gesture in Embodied Communication and Human-Computer Interaction, Stefan Kopp and Ipke Wachsmuth (Eds.). Springer, Berlin, 73--84.
6. Jacob Lowell Bishop and Matthew A. Verleger. 2013. The flipped classroom: A survey of the research. In Proceedings of the American Society for Engineering Education Annual Conference 8 Exposition, Vol. 30.
7. Blender {n.d.}. blender.org - Home of the Blender project - Free and Open 3D Creation Software. Retrieved October 15, 2018 from https://www.blender.org/.
8. Thomas Both. 2013. Design Thinking Bootleg. Retrieved October 15, 2018 from https://dschool.stanford.edu/resources/the-bootcamp-bootleg.
9. Mason Bretan, Sageev Oore, Jesse Engel, Douglas Eck, and Larry Heck. 2017. Deep music: Towards musical dialogue. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI’17). 5081--5082.
10. Jamie Bullock and Ali Momeni. 2015. Ml.lib: Robust, cross-platform, open-source machine learning for Max and PureData. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME’15). 265--270.
11. Christopher J. C. Burges. 1998. A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 2, 2 (1998), 121--167.
12. Antonio Camurri, Shuji Hashimoto, Matteo Ricchetti, Andrea Ricci, Kenji Suzuki, Riccardo Trocca, and Gualtiero Volpe. 2000. EyesWeb: Toward gesture and affect recognition in interactive dance and music systems. Comput. Mus. J. 24, 1 (2000), 57--69.
13. Baptiste Caramiaux, Nicola Montecchio, Atau Tanaka, and Frédéric Bevilacqua. 2014. Adaptive gesture recognition with variation estimation for interactive systems. ACM Trans. Interact. Intell. Syst. 4, 4, Article 18 (Dec. 2014), 34 pages.
14. Alex J. Champandard. 2016. Semantic style transfer and turning two-bit doodles into fine artworks. CoRR abs/1603.01768 (2016). arxiv:1603.01768 http://arxiv.org/abs/1603.01768.
15. Louis Cohen, Lawrence Manion, and Keith Morrison. 2011. Research Methods in Education. Routledge, Abingdon, UK.
16. Luke Dahl, Jorge Herrera, and Carr Wilkerson. 2011. TweetDreams: Making music with the audience and the world using real-time Twitter data. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME’11). 272--275.
17. John Dewey. 1938. Experience 8 Education. Kappa Delta Pi, New York.
18. Charles Dodge. 1970. Earth’s Magnetic Field: Realizations in Computed Electronic Sound. Audio Recording, Nonesuch/Elektra 71250.
19. Daniel C. Edelson, Douglas N. Gordin, and Roy D. Pea. 1999. Addressing the challenges of inquiry-based learning through technology and curriculum design. J. Learn. Sci. 8, 3--4 (1999), 391--450.
20. Jerry Alan Fails and Dan R. Olsen, Jr. 2003. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (IUI’03). ACM, New York, NY, 39--45.
21. Rebecca Fiebrink. 2011. Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. Ph.D. Dissertation, Princeton University.
22. Rebecca Fiebrink. 2016. Machine Learning for Musicians and Artists (online course). Retrieved October 15, 2018 from https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists/info.
23. Rebecca Fiebrink. 2018. The Wekinator: Software for real-time, interactive machine learning. http://www.wekinator.org/.
24. Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 147--156.
25. Rebecca Fiebrink, Dan Trueman, and Perry R. Cook. 2009. A meta-instrument for interactive, on-the-fly machine learning. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME’09). 280--285.
26. The Processing Foundation. {n.d.}. The Processing Foundation. Retrieved April 6, 2018 from https://processingfoundation.org/.
27. Jason Freeman, Brian Magerko, Tom McKlin, Mike Reilly, Justin Permar, Cameron Summers, and Eric Fruchter. 2014. Engaging underrepresented groups in high school introductory computing through computational remixing with EarSketch. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE’14). ACM, New York, NY, 85--90.
28. Yoav Freund and Robert E. Schapire. 1997. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. System Sci. 55, 1 (1997), 119--139.
29. Nicholas Edward Gillian and Joseph A. Paradiso. 2014. The gesture recognition toolkit. J. Mach. Learn. Res. 15, 1 (2014), 3483--3487.
30. Google. 2018. Machine Learning Crash Course with TensorFlow APIs. Retrieved June 25, 2018 from https://developers.google.com/machine-learning/crash-course/.
31. Pamela Lynn Grossman. 1990. The Making of a Teacher: Teacher Knowledge and Teacher Education. Teachers College Press, Teachers College, Columbia University.
32. Idit Ed Harel and Seymour Ed Papert. 1991. Constructionism. Ablex Publishing.
33. Thomas J. Howard, Stephen J. Culley, and Elies Dekoninck. 2008. Describing the creative design process by the integration of engineering design and cognitive psychology literature. Des. Stud. 29, 2 (2008), 160--180.
34. Aleata Hubbard. 2018. Pedagogical content knowledge in computing education: A review of the research literature. Comput. Sci. Educ. 28, 2 (2018), 117--135.
35. Andrew Hugill and Hongji Yang. 2013. The creative turn: New challenges for computing. Int. J. Creat. Comput. 1, 1 (2013), 4--19.
36. Kadenze Inc. {n.d.}. Online STEAM Courses From Top Universities : Kadenze. Retrieved June 15, 2018 from www.kadenze.com.
37. NYU ITP. 2018. ML5js: Friendly Machine Learning for the Web. Retrieved June 15, 2018 from https://ml5js.org/.
38. Giulio Jacucci, Anna Spagnolli, Alessandro Chalambalakis, Ann Morrison, Lassi Liikkanen, Stefano Roveda, and Massimo Bertoncini. 2009. Bodily explorations in space: Social experience of a multimodal art installation. In Proceedings of the International Federation for Information Processing Conference on Human-Computer Interaction. Springer, 62--75.
39. Katy Jordan. 2015. Massive open online course completion rates revisited: Assessment, length and attrition. Int. Rev. Res. Open Distrib. Learn. 16, 3 (2015).
40. René F. Kizilcec, Chris Piech, and Emily Schneider. 2013. Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. In Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK’13). ACM, New York, NY, 170--179.
41. A. J. Ko. 2017. We need to learn how to teach machine learning. Retrieved October 1, 2018 from https://medium.com/bits-and-behavior/we-need-to-learn-how-to-teach-machine-learning-acc78bac3ff8.
42. David A. Kolb. 1984. Experiential Learning: Experience as the Source of Learning and Development. Prentice Hall, Englewood Cliffs, NJ.
43. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097--1105.
44. Ryan Laney. 2017. Syntien: OSC Control Surface. Retrieved from https://itunes.apple.com/us/app/syntien/id1203153534?mt=8.
45. John Lansdown. 1989. Generative techniques in graphical computer art: Some possibilities and practices. In Computers in Art, Design and Animation. Springer, 56--79.
46. Michael Lee, Adrian Freed, and David Wessel. 1991. Real-time neural network processing of gestural and acoustic signals. In Proceedings of the International Computer Music Conference (ICMC’91). 277--280.
47. Beth Logan. 2000. Mel frequency cepstral coefficients for music modeling. In Proceedings of the International Symposium on Music Information Retrieval (ISMIR’00).
48. Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, D. B. Tsai, Manish Amde, Sean Owen, Doris Xin, Reynold Xin, Michael J. Franklin, Reza Zadeh, Matei Zaharia, and Ameet Talwalkar. 2016. MLlib: Machine learning in Apache Spark. J. Mach. Learn. Res. 17, 34 (2016), 1--7.
49. Punya Mishra and Matthew J. Koehler. 2006. Technological pedagogical content knowledge: A framework for teacher knowledge. Teach. Coll. Rec. 108, 6 (2006), 1017.
50. Luis C. Moll, Cathy Amanti, Deborah Neff, and Norma Gonzalez. 1992. Funds of knowledge for teaching: Using a qualitative approach to connect homes and classrooms. Theory Pract. 31, 2 (1992).
51. Alexander Mordvintsev, Christopher Olah, and Mike Tyka. 2015. Inceptionism: Going deeper into neural networks. Retrieved April 6 2018 from https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html.
52. Meinard Müller. 2007. Information Retrieval for Music and Motion. Springer-Verlag, Berlin, 69--84.
53. François Pachet. 2006. Enhancing individual creativity with interactive musical reflexive systems. Musical Creativity, Irène Deliège and Geraint A. Wiggins (Eds.). Taylor 8 Francis Group, London, Chapter 19.
54. Kayur Patel, Naomi Bancroft, Steven M. Drucker, James Fogarty, Andrew J. Ko, and James Landay. 2010. Gestalt: Integrated support for implementation and analysis in machine learning. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST’10). ACM, New York, NY, 37--46.
55. Kayur Dushyant Patel. 2012. Lowering the Barrier to Applying Machine Learning. Ph.D. Dissertation, University of Washington.
56. Roy D. Pea. 1986. Language-independent conceptual “bugs” in novice programming. J. Educ. Comput. Res. 2, 1 (1986), 25--36.
57. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12, October (Oct. 2011), 2825--2830.
58. Kylie Peppler. 2013. STEAM-powered computing education: Using e-textiles to integrate the arts and STEM. Computer 46, 9 (Sep. 2013), 38--43.
59. Libby Plummer. 2017. The next big act out of Abbey Road could be an AI startup. Wired (19 Sep. 2017). Retrieved from https://www.wired.co.uk/article/abbey-road-red-startups-ai-music.
60. Christopher Raphael. 2010. Music plus one and machine learning. In Proceedings of the 27th International Conference on Machine Learning (ICML’10).
61. Mitchel Resnick, Brad Myers, Kumiyo Nakakoji, Ben Shneiderman, Randy Pausch, Ted Selker, and Mike Eisenberg. 2005. Design principles for tools to support creative thinking. In Report of Workshop on Creativity Support Tools. Washington, DC.
62. Stuart Russell and Peter Norvig. 2016. Artificial Intelligence: A Modern Approach (3rd ed.). Pearson, Chapter 18.
63. Rob Schapire. 2014. COS 511: Theoretical Machine Learning. Retrieved June 15, 2018 from https://www.cs.princeton.edu/courses/archive/spring14/cos511/scribe_notes/0204.pdf.
64. Janelle Shane. 2018. We asked a neural network to bake us a cake. The results were…interesting. Pop. Sci. (29 Mar. 2018). Retrieved from https://www.popsci.com/neural-network-bakes-a-cake.
65. Lee S. Shulman. 1986. Those who understand: Knowledge growth in teaching. Educ. Res. 15, 2 (1986), 4--14.
66. Tom Simonite. 2017. A ‘neurographer’ puts the art in artificial intelligence. Wired (6 Jul. 2017). Retrieved from https://www.wired.com/story/neurographer-puts-the-art-in-artificial-intelligence/.
67. Laetitia Sonami. 1991. Lady’s Glove. Retrieved June 15, 2018 from http://sonami.net/ladys-glove/.
68. Juha Sorva. 2013. Notional machines and introductory programming education. Trans. Comput. Educ. 13, 2, Article 8 (Jul. 2013), 31 pages.
69. TechSpark. 2017. Guest blog: How one UWE Bristol researcher’s work led to AI gloves that control sound. Retrieved June 15, 2018 from https://techspark.co/guest-blog-uwe-bristols-ai-gloves-control-sound-without-need-laptop-mouse/.
70. Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016. WaveNet: A generative model for raw audio. CoRR abs/1609.03499 (2016). arxiv:1609.03499 http://arxiv.org/abs/1609.03499.
71. Matthew Wright and Adrian Freed. 1997. Open sound control: A new protocol for communicating with sound synthesizers. In Proceedings of the International Computer Music Conference (ICMC’97).
72. Matthew John Yee-King, Mick Grierson, and Mark d’Inverno. 2017. Evidencing the value of inquiry based, constructionist learning for student coders. Int. J. Eng. Pedag. 7, 3 (2017), 109--129.

Index Terms

• Applied computing

  • Arts and humanities

• Computing methodologies

  • Machine learning

• Social and professional topics

  • Professional topics

    • Computing education