NSF funding to Fermüller, Muresanu, Shamma for musical instrument distance learning using AI

ISR-affiliated Research Scientist Cornelia Fermüller (UMIACS) is the principal investigator and Professor Shihab Shamma (ECE/ISR) and Associate Professor Irina Muresanu (ARHU/UMIACS) are the co-PIs for “Emphasizing Explanation in AI Augmented String Instrumental Education.” The three-year, $900K grant from the National Science Foundation furthers Fermüller and Muresanu’s ongoing work on violin performance instruction using AI and distance learning (see “Work leading to the NSF grant” below).

Learning to play a musical instrument is a lengthy, costly and complex process. New online education and AI tools could provide music education for a much more diverse student population. However, researchers are only at the beginning of understanding how to remotely teach musical instrument skills, including how to keep students engaged in a discipline that involves individual motor instructions.

Muresanu (violin pedagogy) and Fermüller and Shamma (visual and auditory signal analysis) are developing an AI platform called “VAIolin” to analyze a student’s playing during individual practice time. Cameras and microphones on handheld devices provide feedback on posture, bow movements, sound quality and selection of curricular materials.

The researchers are building a feedback system to provide advice based on visual and auditory analysis and causal relationships of errors. They are also assigning educational materials specific to the students based on the perceptual evaluation and a library of recorded materials. The combination of developed AI software, data collected through observation of violin students, and the music library is a major step forward for the field of music pedagogy and the innovation of tools for studying human motor learning through perception.

Work leading to the NSF grant

In February 2023, Fermüller and Muresanu received a University of Maryland Grand Challenges Team Project Grant, “Music Education for All through AI.” A news story on their work was featured in Maryland Today. It was accompanied by the video below, which was produced by Maria Herd of UMIACS.