Imagine listening to a live jazz improvisation. Moved by what you hear, you decide to try to reproduce the solo on your piano at home. To do so, you would have had to record the piece. Then you would have to listen to it over and over again, and transcribe the notes yourself.
It’s a painstaking process. That’s one reason why Zhiyao Duan, an assistant professor of electrical and computer engineering at the Hajim School, his PhD student Andrea Cogliati, and David Temperley, a professor of music theory at Eastman, have been developing a computer program that could receive audio, extract data from it, and produce an accurate printed musical score.
This summer they’ve have two additional researchers working on the project. Two undergraduate students—Arlen Fan ’18, an electrical and computer engineering major, and rising senior Andrew Smith, a computer science major and music minor at the University of Central Florida—joined the project as part of the NSF-funded Research Experience for Undergraduates (REU) program “Computational Methods for Understanding Music, Media, and Minds.”
According to Duan, “Machine transcription is fast, but it isn’t as accurate as a person. Human transcription is accurate, but not very fast. We want to combine these to have both an accurate and fast system.”
Fan and Smith have spent the summer creating an interface that would allow human users to fix portions of the transcription that the computer notated incorrectly and then input these changes into the computer. Using machine learning techniques, the computer would then “learn” from these mistakes in order to avoid similar errors in the future.
Such an automatic music transcription program would help advance music education, retrieval, and research. For example, it would allow users to notate music for which a score is not readily available or to search for music with similar melodies and chord progressions.
“There are musical traditions—not just jazz, but types of Eastern or Indian music—where there is not a written score,” Cogliati says. “The program can bridge the gap between oral tradition and written notation.”
Smith says he is already interested in using the program to produce sheet music for improvisational musicians such as 20th-century comedian and pianist Victor Borge because “you can’t find his sheet music anywhere.”
Reflecting on his decision to work at Rochester this summer, Smith says, “I’m interested in music theory and performance, as well as the technical aspects of computers, so this opportunity fit like a glove.”
Now he’s considering pursuing a graduate degree at Rochester.“Now that I’ve worked in this lab and got to experience how interesting this project is, I might try to apply here next year.”
