News / #music Tag Music 370 articles archived under #music · RSS Sign in to follow r/LocalLLaMA community 2mo ago Anyone gotten Gemma 4 12B (unified audio) to actually attend to speech with a large system prompt? I'm trying to use Gemma 4 12B — the new encoder-free unified model (audio/vision/text in one) — for a one-pass audio → response voice assistant: feed the recorded WAV + system prompt and get the reply back as text directly, collapsing the separate ASR + LLM steps into a single… 31 arXiv — NLP / Computation & Language research 2mo ago SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech arXiv:2606.06037v2 Announce Type: cross Abstract: Large audio language models (LALMs) are increasingly deployed in real-world applications, yet their safety alignment is still primarily evaluated on monolingual, text-based harmful prompts. This leaves their generalizability… 29 arXiv — NLP / Computation & Language research 2mo ago CANVAS: Captioning Art with Narrative Visual-Audio AI Systems arXiv:2606.09846v1 Announce Type: cross Abstract: Visual art remains largely inaccessible to blind and low-vision (BLV) audiences due to brief or absent alt-text, which rarely conveys the sensory, spatial, or emotional qualities of an artwork. This study presents an automated… 6 arXiv — NLP / Computation & Language research 2mo ago From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs arXiv:2606.10147v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) can listen and see, but how do audio and visual signals actually travel through the network to shape an answer? Despite their growing role in research and real-world applications, the… 30 Google DeepMind official-blog 2mo ago Fluid, natural voice translation with Gemini 3.5 Live Translate Gemini 3.5 Live Translate brings near real-time, natural speech translation to Google AI Studio, Google Translate and Google Meet. 32 Hugging Face Daily Papers research 2mo ago Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders Abstract Research demonstrates that hallucinations in Whisper ASR can be detected and reduced using internal representations from audio encoder activations and Sparse AutoEncoder latents, achieving significant hallucination rate reduction with minimal speech transcription… 20 Hugging Face Daily Papers research 2mo ago EMMA: Extracting Multiple physical parameters from Multimodal Data Abstract EMMA is a physics-informed multimodal framework that directly recovers dynamical parameters from raw video, audio, and image data using a Liquid Time-Constant network and physics-constrained loss. Generated by Qwen/Qwen2.5-Coder-32B-Instruct We introduce EMMA, a… 33 llama.cpp releases dev-tools 2mo ago b9555 metal : fix im2col 1D case (audio models) ( #24220 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64… 29 Hugging Face Daily Papers research 2mo ago Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development Abstract Confidence-based loss weighting via entropy-derived log-barrier enables improved audio generation through adaptive gradient scaling in supervised diffusion training. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Confidence-based loss weighting is usually avoided in… 36 Hugging Face Daily Papers research 2mo ago MMAE: A Massive Multitask Audio Editing Benchmark Abstract MMAE presents a comprehensive benchmark for instruction-based audio editing across multiple modalities and complexity levels, revealing significant gaps in current model capabilities. Generated by Qwen/Qwen2.5-Coder-32B-Instruct We introduce MMAE, a Massive Multitask… 24 arXiv — Machine Learning research 2mo ago Making the Most of Limited Data: Score-Aware Training for Text-to-Music Generation arXiv:2606.07387v1 Announce Type: new Abstract: State-of-the-art text-to-music generation systems rely on massive proprietary datasets and industrial-scale compute, making it impossible to disentangle architectural contributions from resource advantages. We propose… 15 arXiv — NLP / Computation & Language research 2mo ago HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec arXiv:2606.06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models. New codec architectures with semantic and acoustic disentanglement have emerged. There are two main… 21 arXiv — NLP / Computation & Language research 2mo ago MMAE: A Massive Multitask Audio Editing Benchmark arXiv:2606.07229v1 Announce Type: cross Abstract: We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation,… 8 arXiv — NLP / Computation & Language research 2mo ago Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition arXiv:2606.07309v1 Announce Type: cross Abstract: Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available. We study this question… 14 arXiv — NLP / Computation & Language research 2mo ago DirectAudioEdit: Inversion-Free Text-Guided Audio Editing via Diffusion Prediction Contrast arXiv:2606.07356v1 Announce Type: cross Abstract: Text-guided audio editing aims to modify the language-specified acoustic content while preserving edit-irrelevant source components. Existing training-free methods typically rely on inversion-based editing. While inversion-free… 25 r/LocalLLaMA community 2mo ago Dockerized Nemotron 3.5 ASR — Switched from Parakeet, better multilingual support + streaming (4.5x realtime speed on cpu) I was originally using Parakeet for my speech recognition pipeline but decided to give Nemotron 3.5 a shot. After testing it on some multilingual audio clips, it's been working great so far. What sold me: - Better language support (40+ locales from one model) - Native streaming… 17 r/LocalLLaMA community 2mo ago Gemma4 12B - Experiences? Anyone check out the new Gemma4 12B that dropped 3 days ago? Integrated vision and audio recognition, no mmpro needed plus tool use. Q4 quant is like 8gb RAM. Crazy fast and great quality for it's size. No, it's not as good as a 27B or 31B. But it's damn close. Curious what… 24 r/LocalLLaMA community 2mo ago Best Coding Harness for Qwen3.6 35B? I've been happily using GitHub Copilot for 7-8 months, primarily in Visual Studio and VS Code, mostly with the built-in flagship models and have felt like the output is worth the cost. Lately I've been playing with a lot of different local LLM models and decided to try using… 32 r/LocalLLaMA community 2mo ago I just realized how good MoE models are for consumer hardware I've been tinkering around with LLM for a while now, started with LM Studio like probably all of us and wanted to go into headless selhosted model so that I can use my macbook and still use my AI models. I've been using Qwen 3.6 (and 3.5) 27B on my main computer which has a… 7 r/MachineLearning community 2mo ago Benchmark: ONNX Runtime vs HF Transformers vs GGUF for Parakeet TDT 0.6B on CPU-only hardware [D] Sharing a small CPU inference benchmark for nvidia/parakeet-tdt-0.6b-v3 that turned up a result I didn't expect going in. Setup: 2 x86-64 vCPUs (AVX2/FMA), 7.7GB RAM, no GPU. Test audio: 16.78s Harvard sentences at 16kHz mono. Results: Inference path RTF Peak Memory CPU… 26 r/LocalLLaMA community 2mo ago Benchmark & Reality Check on Gemma 4 12B: Great model, but your local settings are probably breaking it (Fix inside) I completed a Python bug hunting benchmark with Gemma 4 12B. I used the Unsloth Dynamic Q5 GGUF model. The model has good capabilities. Default settings in LM Studio disable the reasoning. Fix the LM Studio reasoning configuration. LM Studio looks for Qwen tokens. Gemma 4 uses… 30 Hugging Face Daily Papers research 2mo ago Multimodal Music Recommendation System using LLMs Abstract A multimodal framework for session-based music recommendation integrates audio, lyric, and semantic signals with LLM-based sequential reasoning to improve recommendation accuracy. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Music recommendation systems typically treat… 16 arXiv — NLP / Computation & Language research 2mo ago MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models arXiv:2606.05177v1 Announce Type: new Abstract: Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text. We introduce MCBench, a benchmark with 1196 scenarios spanning four… 5 arXiv — NLP / Computation & Language research 2mo ago Forgive or forget: Understanding the context of hate in audio retrieval systems arXiv:2606.05857v1 Announce Type: new Abstract: Handling toxic retrieval in text-to-audio systems is challenging due to contextual dependencies. Existing strategies (e.g., rephrasing, summarization) risk altering intent or omitting details. We propose a post hoc causal debiasing… 27 arXiv — NLP / Computation & Language research 2mo ago To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection arXiv:2606.05931v1 Announce Type: new Abstract: When retrieving a person from a video archive by voice and face, should the system be multimodal or not? In real-world broadcast archives, unlike curated benchmarks, a target may be heard but unseen, seen but unheard, or both.… 21 r/LocalLLaMA community 2mo ago Higgs Audio v3 TTS 4B. Built for voice chat. Support 100 languages and inline control.   submitted by   /u/FerretLegitimate6929 [link]   [comments] 31 llama.cpp releases dev-tools 2mo ago b9503 fix(mtmd): handle Gemma 4 audio projector embedding size ( #24091 ) mtmd: handle Gemma 4 audio projector embedding size rm projection_dim from clip_n_mmproj_embd Co-authored-by: Xuan Son Nguyen [email protected] macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64,… 28 arXiv — NLP / Computation & Language research 2mo ago DetectZoo: A Unified Toolkit for AI-Generated Content Detection Across Text, Audio, and Image Modalities arXiv:2606.04205v1 Announce Type: cross Abstract: The growing popularity and capacity of generative models have eroded the distinction between human and machine-generated content, motivating a growing body of work on detection across text, images, and audio. Most available… 6 arXiv — NLP / Computation & Language research 2mo ago CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding arXiv:2606.04418v1 Announce Type: cross Abstract: Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency,… 6 Hugging Face Daily Papers research 2mo ago Audio Interaction Model Abstract A unified streaming audio model is developed that combines offline task execution with real-time audio instruction following through an end-to-end framework supporting multiple audio interaction capabilities. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Audio is an… 20 r/LocalLLaMA community 2mo ago How to use audio and vision modalities in llama.cpp? How to use audio and vision modalities in llama.cpp with Gemma4 12B it? I’m on release b9494, but when I run llama-cli it shows “modalities: text” only, and crashes if I try to add an image.   submitted by   /u/No-Leave-4512 [link]   [comments] 20 r/LocalLLaMA community 2mo ago Best way to index full Italian Wikipedia for 100% offline RAG in LM Studio? Hi everyone, I want to set up a 100% offline RAG system using LM Studio and the entire Italian Wikipedia (text-only, no images). My goal is to index the database once so my local LLMs can query it for up-to-date factual knowledge without internet access. Here are my PC specs:… 14 r/LocalLLaMA community 2mo ago google/gemma-4-12B · Hugging Face Gemma is a family of open models built by Google DeepMind. Gemma 4 models are multimodal, handling text and image input (with audio supported on E2B, E4B, and 12B) and generating text output. This release includes open-weights models in both pre-trained and instruction-tuned… 29 Hugging Face Daily Papers research 2mo ago MERIT: Learning Disentangled Music Representations for Audio Similarity Abstract MERIT framework learns disentangled music representations for melody, rhythm, and timbre through conditional audio generation and source-separated stems, enabling nuanced musical queries. Generated by Qwen/Qwen2.5-Coder-32B-Instruct Current music similarity models… 21 Vercel — AI dev-tools 2mo ago Grok Imagine Video 1.5 on AI Gateway Grok Imagine Video 1.5 from xAI is now available on AI Gateway. The model generates video from an input image with synchronized audio in a single pass. This release improves audio quality, prompt following, and photorealism. Face accuracy and character consistency are stronger… 26 r/LocalLLaMA community 2mo ago Benchmarks of 20 small LLMs on a 6GB RTX 4050 I'm looking for models that can run on my GPU and actually do something useful. I think that any small difference could be a "big" improvement, because they are all so small. So I went to the LM studio database and searched many variants from the same family, trying to select… 37 r/LocalLLaMA community 2mo ago NVIDIA releases Cosmos 3 Omnimodal world modelson HF https://huggingface.co/nvidia/Cosmos3-Super-Text2Image Nano: 16B Super: 64B Cosmos3 is a collection of Omnimodal world models capable of generating dynamic, high-quality video, image, audio, and action commands from combinations of text, image, video, and action trajectory… 7 r/LocalLLaMA community 2mo ago Moss tts 1.5 8b Examples. It is the currently best voice cloning model for English as of June 2026 Moss tts 1.5 8b is better than fish audio s2 pro and qwen 3 tts voice clone tts. You can easily get more better quality if you set up the duration of the voice in output you want and some temperature and other changes. This was just used on default setting. It can be improved… 20 arXiv — NLP / Computation & Language research 2mo ago Sandboxed Coding Agents are Competitive Omni-modal Task Solvers arXiv:2606.00579v1 Announce Type: new Abstract: As multimodal LLMs increasingly target video and audio, it is often assumed that such tasks require native omnimodal models. We show that this is not always the case: coding agents with only text+image access and a sandboxed… 37 Hugging Face Daily Papers research 2mo ago StreamChar: Long-Horizon Streaming Character Audio-Video Generation with Decoupled Orchestration Abstract StreamChar enables real-time streaming audio-video generation for character animation by separating long-horizon orchestration from short-window denoising through an LLM-based orchestrator and joint audio-video DiT, achieving efficient deployment via two-stage… 8 Smol AI News news-outlet 2mo ago not much happened today **NVIDIA** led open-source AI model releases with **Cosmos 3**, a comprehensive omnimodal world model unifying language, image, video, audio, and action using a Mixture-of-Transformers design, and **Nemotron 3 Ultra**, a **550B** parameter open-weight model noted for high… 33 Hugging Face Daily Papers research 2mo ago Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer Abstract SwanSphere presents a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts using causal autoregressive diffusion transformers and multimodal learning strategies. AI-generated summary Real-time and accurate spatial… 25 r/LocalLLaMA community 2mo ago Llama Studio v0.2.0 I have made an update to my llama-server WebUI based on some awesome feedback and interaction with the community. 1) JSON model config replaced by per-model shell scripts. Run from CLI, paste from unsloth, email to your buddy or post to reddit: Using real shell scripts to store… 17 r/LocalLLaMA community 2mo ago <Think> toggle button for llama.cp web chat for QWEN3.6 https://preview.redd.it/od6suf6j7g4h1.png?width=619&format=png&auto=webp&s=d31fb903ea68f58e3a641bfd275d59eeb5cce445 Missing a button in llama-serve webchat to toggle reasoning on/off like in LM Studio? This is a snippet that runs in https://www.tampermonkey.net/ a browser… 34 r/LocalLLaMA community 2mo ago Open source : Turning vocal imitations into sound effects. (New UX for sound generation) Hello guys I want to introduce my new project! Have you ever needed a specific sound while making a video or a game? You know exactly what it sounds like in your head, but have no idea how to search for it. That’s why sound design meetings at game studios often turn into people… 12 r/LocalLLaMA community 2mo ago this new Moss tts 1.5 is damn good with voice cloning https://huggingface.co/spaces/OpenMOSS-Team/MOSS-TTS-v1.5 I prefer this over fish audio s2 pro because fish audio dont allow commercial use Long Cat DiT 3.5 is also a another good model.   submitted by   /u/9r4n4y [link]   [comments] 38 r/LocalLLaMA community 2mo ago I compared all specs of the major GPUs/machines that are being used here, because bandwidth is not everything. Some of ya'll need a reality check. Hot takes: - Mac studio is overpriced Raspberry Pi that is way more inefficient than people think (together with most macs). M5 MBP is better with the "tensor" MMA, but not by much. - Spark was actually decent when it was just 3-4k. Strix is obviously much better now - 3090 are… 26 r/LocalLLaMA community 2mo ago Unsloth Studio updated to support training with MLX on macs The title says it all. I noticed this morning when reviewing Unsloth Studio github that training with MLX is now fully supported. Not sure when this was added but must have been within the last couple of weeks since last I checked it said "coming soon." I haven't personally… 36 Hugging Face Daily Papers research 2mo ago ChildVox: A Speech, Audio, and Large Audio-Language Model Benchmark in Understanding and Characterizing Sound across Childhood Abstract ChildVox presents a comprehensive benchmark for analyzing children's acoustic communication across developmental stages using diverse audio and speech models. AI-generated summary We present ChildVox, a novel benchmark for characterizing the diverse acoustic signals… 15 arXiv — Machine Learning research 2mo ago Auditing Training Data in Generative Music Models via Black-Box Membership Inference arXiv:2605.29202v1 Announce Type: new Abstract: Recent advances in text-to-music generation enable high-fidelity synthesis of structured musical audio, raising growing concerns about data provenance, consent, and training transparency. These models are typically trained on… 29 Page 6 of 8 · 370 articles ← Newer Older →
