'AI artists' perform improvised music at Jarasum Jazz Festival

Jazz pianist Heo Dae-uk, left, and artificial intelligence (AI) pianist Scott perform during the 23rd Jarasum Jazz Festival, held in Gapyeong County, Gyeonggi Province, from Oct. 9 to 11. Courtesy of Korea Advanced Institute of Science and Technology

Artificial intelligence (AI) models developed by a research team led by Korea Advanced Institute of Science and Technology (KAIST) professor Nam Ju-han made their debut at one of Korea’s leading jazz festivals, performing improvised jazz alongside human artists.

According to the university Friday, the team showcased human-AI music collaboration technology that analyzes the flow of human performances in real time, allowing AI to take over performances or generate improvised accompaniment during the 23rd Jarasum Jazz Festival, held in Gapyeong County, Gyeonggi Province, from Oct. 9 to 11.

A total of 12 human jazz artists participated in the event, collaborating with AI in 26 pieces across 12 performances.

The technology is different from existing music AI services as it enables collaboration with human musicians in jazz, a genre characterized by improvisation.

Jazz musicians often change the tempo or create new melodies based on how they feel in the moment. Because the same piece can be performed differently each time, musicians must listen to one another and adjust their playing accordingly. For AI to participate in jazz performances, it must go beyond playing precomposed music and be able to listen to human musicians and respond to changes in their performances.

A screen shows the augmented reality character of artificial intelligence pianist Scott during a performance with jazz pianist Heo Dae-uk during the 23rd Jarasum Jazz Festival, held in Gapyeong County, Gyeonggi Province, from Oct. 9 to 11. Courtesy of Korea Advanced Institute of Science and Technology

The team showcased two AI models — a pianist model called Scott and accompanist DeepPocket.

Scott compared human musicians' performances with sheet music in real time to take over at appropriate moments and generate new solos reflecting the melodies and rhythms of the musicians.

The music generated by Scott was transmitted to a Yamaha Disklavier acoustic reproducing piano, which played the music through actual key and pedal movements. On stage, an augmented reality character was overlaid on live camera footage using movement data generated by technology that simulates a pianist's hand movements.

DeepPocket analyzed the work of human musicians and generated piano, bass and drum accompaniment in real time, adjusting to their notes, rhythms and tempo. It produced music alongside pianists and guitarists without any prior preparation, continuously adapting to their playing style.

In 2023, the research team showcased AI technology that adjusted the timing of piano accompaniment during a performance by soprano Sumi Jo. This time, the team took the technology a step further, enabling AI to generate its own solos and accompaniment on a jazz stage.

“The performances demonstrated that AI can listen to human musicians and create music alongside them by responding to the flow of their performances, marking a significant advance from simply playing predefined music,” Nam said. “We will continue our research to develop AI into a musical partner that can exchange creative inspiration with human musicians, rather than simply serving as a tool.”