A new program developed by Rochester researchers uses machine learning to allow users to improvise duets with an artificial intelligence partner.
To play a duet with German composer Johann Sebastian Bach, you don’t have to travel back to the 18th century; thanks to a new program developed by researchers at the University of Rochester, you only need a computer.
The web-based program, called BachDuet, was developed by Zhiyao Duan, an associate professor of electrical and computer engineering and of computer science, and members of his lab, including Yongyi Zang ’23 and PhD student Christodoulos Benetatos. BachDuet allows a person to improvise duets in the style of Bach with an artificial intelligence (AI) counterpoint in real time. By visiting the BachDuet website, a user can play duets with the AI agent using a computer keyboard, mouse, touchscreen, or MIDI keyboard.
In BachDuet, the AI agent will either follow the human musician, lead the improvisation, or “just goof around,” Duan says.
But the researchers foresee the program could have applications in music education, entertainment, and music therapy. For instance, says Duan, “most of the existing musical therapies are listening-based, but in recent years we have seen some playing-based music therapies that have been shown to help improve cognitive functions.”
BachDuet is part of Duan’s research project, funded by a National Science Foundation Faculty Early Career Development (CAREER) award, to develop algorithms and systems that allow humans to work with machines to harmonize pitch, coordinate musical timing, and convey expression in music.
His lab uses signal processing, artificial intelligence, and machine learning algorithms, which can empower machines with music perception skills, more expressive performance, and deeper “understanding” of music theory and composition. For this project, the team input hundreds of Bach’s compositions and used machine learning techniques to allow the program to “study” the music. During improvisation, the program assumes a 4/4 time signature and makes a decision about which note to trigger or sustain.
“The goal of our project is to develop tools that allow humans to play music with computer systems in a way similar to human musicians collaborating with each other,” Duan says. “With BachDuet, we wanted to develop a tool that allows musicians and novices alike to practice counterpoint improvisation.”
“And,” he adds, “to have fun.”
Read more
CAREER awards spur junior researchers along varied paths
Four Rochester researchers from four different fields are 2019 recipients of the National Science Foundation’s most prestigious recognition for junior faculty members.
Giving virtual reality a ‘visceral’ sound
Using recital halls as their “labs,” and recording some of the best music students in the world, University researchers are creating immersive virtual reality concert videos.
The mysteries of music—and the key of data
Researchers at the University are at the cutting edge of the intersection of data science and music: developing databases to study music history and creating algorithms to automatically identify a genre or singer.
