Infinite Music Glitch on my Arduino with Magenta Realtime 2

I built a local voice AI realtime music setup where my ESP32 microcontroller talks to my MacBook over WebSockets. The microcontroller is just a tiny Arduino-based device with a mic and speaker, and the MacBook M4 Pro runs Magenta Realtime 2 locally and streams the audio back to the device.

The fun part is that it’s agentic and conversational. So I can tap the ESP32, speak into it, and it uses MLX Whisper to transcribe what I said. Then after detecting VAD, it sends that to a Qwen model, which decides what tool call to make, like adding drums, making the music Lo-fi, adding Jazz bebop, removing guitar, or changing the instruments in the music.

GitHub link: https://github.com/akdeb/jambox

HF link: https://huggingface.co/google/magenta-realtime-2