NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Mirrored from arXiv — NLP / Computation & Language for archival readability. Support the source by reading on the original site.
Computer Science > Computation and Language
arXiv:2609.21967 (cs)
[Submitted on 18 Sep 2026]
Title:NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Authors:Jagadeesh Balam, Travis Bartley, Edresson Casanova, Sanjay Chauhan, Chen Chen, Zhehuai Chen, Zijia Chen, Francesco Ciannella, Slyne Deng, Mikyas Desta, Harishchandra Dubey, Slim Essid, Nourchene Ferchichi, Boris Ginsburg, Mariana Graterol Fuenmayor, Negar Habibi, Kevin Hu, Anand Joseph, Viraj Karandikar, Myungjong Kim, Viacheslav Klimkov, Seelan Lakshmi Narasimhan, Lily Lee, Jason Li, Eileen Long, Ameya Mahabaleshwarkar, Aditya Malte, Adi Margolin, Sasha Meister, Valentin Mendelev, Oluwatobi Olabiyi, Ankita Pasad, Yifan Peng, Elena Rastorgueva, Jayda Ritchie, Jason Roche, Nikhil Srihari, Yuanhang Su, Yoshi Suhara, Viet Anh Trinh, Jinhan Wang, Piotr Zelasko, Hui Wang, Puhui Meng, Chaosen Zhang, Yunsheng Liu, Shawn Wang, Wenjing Li, Zhonglei He
View a PDF of the paper titled NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities, by Jagadeesh Balam and Travis Bartley and Edresson Casanova and Sanjay Chauhan and Chen Chen and Zhehuai Chen and Zijia Chen and Francesco Ciannella and Slyne Deng and Mikyas Desta and Harishchandra Dubey and Slim Essid and Nourchene Ferchichi and Boris Ginsburg and Mariana Graterol Fuenmayor and Negar Habibi and Kevin Hu and Anand Joseph and Viraj Karandikar and Myungjong Kim and Viacheslav Klimkov and Seelan Lakshmi Narasimhan and Lily Lee and Jason Li and Eileen Long and Ameya Mahabaleshwarkar and Aditya Malte and Adi Margolin and Sasha Meister and Valentin Mendelev and Oluwatobi Olabiyi and Ankita Pasad and Yifan Peng and Elena Rastorgueva and Jayda Ritchie and Jason Roche and Nikhil Srihari and Yuanhang Su and Yoshi Suhara and Viet Anh Trinh and Jinhan Wang and Piotr Zelasko and Hui Wang and Puhui Meng and Chaosen Zhang and Yunsheng Liu and Shawn Wang and Wenjing Li and Zhonglei He
View PDF
HTML (experimental)
Abstract:We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combines a streaming speech encoder and decoder-only language model with parallel specialized output streams for agent text and structured function calls, an auxiliary RNN-T branch for incremental user transcription, and a streaming TTS decoder. This design enables the model to listen, transcribe, reason, invoke tools, and speak within a unified streaming architecture while preserving the temporal behavior required for natural conversation. On Full-Duplex-Bench 1.0, NemotronLabs VoiceChat achieves the lowest pause-handling takeover rates among evaluated open-weight systems, 100\% takeover following user interruptions, and a 4.33/5 post-interruption response-quality score. On Full-Duplex-Bench 1.5, it resumes its response after user backchannels in 93\% of cases. NemotronLabs VoiceChat obtains a 55.1 normalized average on VoiceBench and, on Full-Duplex-Bench 3.0 (FDB 3.0), achieves 82.5\% tool-selection F1, while argument accuracy and end-to-end tool execution remain areas for improvement. These results demonstrate that full-duplex interaction, speech recognition and generation, general language capabilities, and external tool use can be integrated in a single open speech-to-speech model without sacrificing real-time conversational behavior.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.21967 [cs.CL] |
| (or arXiv:2609.21967v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.21967
arXiv-issued DOI via DataCite (pending registration)
|
Full-text links:
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
View a PDF of the paper titled NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities, by Jagadeesh Balam and Travis Bartley and Edresson Casanova and Sanjay Chauhan and Chen Chen and Zhehuai Chen and Zijia Chen and Francesco Ciannella and Slyne Deng and Mikyas Desta and Harishchandra Dubey and Slim Essid and Nourchene Ferchichi and Boris Ginsburg and Mariana Graterol Fuenmayor and Negar Habibi and Kevin Hu and Anand Joseph and Viraj Karandikar and Myungjong Kim and Viacheslav Klimkov and Seelan Lakshmi Narasimhan and Lily Lee and Jason Li and Eileen Long and Ameya Mahabaleshwarkar and Aditya Malte and Adi Margolin and Sasha Meister and Valentin Mendelev and Oluwatobi Olabiyi and Ankita Pasad and Yifan Peng and Elena Rastorgueva and Jayda Ritchie and Jason Roche and Nikhil Srihari and Yuanhang Su and Yoshi Suhara and Viet Anh Trinh and Jinhan Wang and Piotr Zelasko and Hui Wang and Puhui Meng and Chaosen Zhang and Yunsheng Liu and Shawn Wang and Wenjing Li and Zhonglei He
References & Citations
Loading...
Bookmark
Bibliographic Tools
Code, Data, Media
Demos
Related Papers
About arXivLabs
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
More from arXiv — NLP / Computation & Language
-
A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID
Sep 28
-
Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling
Sep 28
-
Not All Memories Are Equal: Hierarchical Collaborative Memory for Validity-Aware Retrieval in LLM Agents
Sep 28
-
Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
Sep 28
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.