Open Source
Open Source

Whisper's rival is here: NVIDIA tops the open ASR board

NVIDIA open-weighted Nemotron 3.5 Transcribe on October 28 (repo name 30B-A4B; per the model card 34B total parameters including the audio encoder with only about 4.3B active per step, on a Lightning 30B-A3B language backbone): transcription in 40 languages, translation across 31, speaker attribution in 13, one-hour audio with up to 8 speakers, and a batch-only offline design with no streaming (a separate 0.6B sibling handles streaming). Two official accuracy figures coexist: 4.71 percent mean English WER on the model card's nine public sets (versus Qwen3-ASR-1.7B at 4.94) and 5.33 on the leaderboard site (private held-out sets added since October 9); Artificial Analysis independently measured 2.81 percent, close to Voxtral Small's 2.77 despite activating roughly a sixth of the parameters. The OpenMDW-1.1 license (a Linux Foundation permissive license) allows free commercial use, redistribution and fine-tuning; weights ship in BF16/FP8/NVFP4/GGUF, with GGUF targeting NeMo-Speech.cpp though the launch blog still called quantized versions coming soon — weights posted does not mean local tooling ready. Training data Granary v2 lists 463,062 public-manifest hours across 48 languages (about 840k fully processed is a separate figure). Division of labor with the site's Hy-SimulMT interpretation piece: transcription is dictation, not real-time translation.

Published October 29, 202611 min read
<!-- nemotron-3-5-transcribe-opensource | open-source | Whisper's rival is here: NVIDIA tops the open ASR board -->

One Hour of Audio, Eight Speakers: The Weights Are Out

Turning a one-hour meeting recording into a clean transcript — who said what, terms spelled right — used to mean manual work or an error-stacking cascade. On October 28, NVIDIA open-weighted its answer: Nemotron 3.5 Transcribe, live on Hugging Face and ModelScope. Here is the repository: NVIDIA-Nemotron-3.5-Transcribe-30B-A4B. Repository: huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B.

Three qualifiers first. It is a transcription model: the model card and NVIDIA research page define it as a promptable offline speech-audio model on the SALM (Speech-Augmented Language Model) architecture, for turning existing recordings into text. Second, it is a MoE: per the card, 34B total parameters including the audio encoder, only about 4.3B active per token — why one GPU can carry it. Third, it does not stream: NVIDIA states internal audio chunking is not a streaming service; it processes recordings that already exist, not live captions.

A scope note: our previous piece covered Tencent's open simultaneous interpretation model — listening and translating live. This piece covers the other job: one hour of eight-speaker audio, turned into a document that keeps every speaker straight. Transcription is not interpretation.

The License: OpenMDW-1.1, Commercial Use in Writing

Commercial safety is a question for the license field, not the headlines. Both the model card's License field and the NVIDIA research page agree: OpenMDW-1.1, officially "ready for commercial or non-commercial use" — introduced in 2025 by the Linux Foundation and the PyTorch Foundation for ML model distribution, and permissive.

Per the standard OpenMDW-1.1 text (per NVIDIA-family repositories): a free, unrestricted grant over the Model Materials covering copyright, patents, database rights, and trade secrets; redistribution requires keeping the license copy and attribution notices; initiating patent or copyright litigation terminates the grant, defensive counterclaims excepted; no restrictions attach to model outputs; and the model ships "AS IS," without warranty. That clause lands squarely on transcription: whether a recording was made legally, and whether participants consented, is the user's responsibility.

Bottom line: commercial use, redistribution, and fine-tuning are permitted without extra authorization — confirmed by the model card, the research page, and independent outlets.

30B in the Name, 34B on the Card: One Sentence Reconciles Both

The repository is named NVIDIA-Nemotron-3.5-Transcribe-30B-A4B, yet the model card reports 34B total parameters. No contradiction: the language backbone is Nemotron 3.5 Lightning 30B-A3B — a hybrid Mamba-2 and Transformer MoE, 30B total and 3B activated; with the speech-side components attached, the card's figure rises to 34B total, about 4.3B active per token. The repo name inherited the backbone's naming; the card reports the full assembly.

Per the research page, the speech side is a Parallel Expert Encoder design: a 600M-parameter Transformer ASR encoder, a frozen ~99M-parameter speaker encoder called Nemotron-3-Diarization, and a CTC word-timestamp head, all feeding the backbone in parallel. Notably, speaker labels are produced during decoding — Sortformer cues enter the language model with the acoustic features — not patched on afterward. NVIDIA claims this is the first SALM to beat dedicated cascade systems on diarized ASR with same-speaker overlap.

Boundaries: 65,536-token context, one hour of audio per recording, 16 kHz mono input, WAV/FLAC/MP3/OGG supported. Word-level timestamps come from a lightweight CTC adapter applied after decoding, aligning any generated transcript back to the audio — long recordings and heavily overlapped speech included.

Promptable Transcription: Same Weights, Different Instructions

The name undersells the model card's capability matrix: transcription in 40 languages, translation to and from English in 31, speaker attribution in 13, up to 8 speakers per recording. The same checkpoint switches tasks on natural-language instructions — "transcribe and label speakers," "translate into English," "what decisions were made in this meeting," "output a PII-redacted transcript." Style is promptable too: verbatim or cleaned-up register, filler words kept or dropped, number formatting.

For terminology-sensitive work, the context word list matters: hand the model an expected-entity list and named-entity errors drop to roughly one-sixth (the official "six-fold" figure, from ContextASR-Bench). The official blog's examples are HbA1c and metformin — in medicine, misspelling a drug name is expensive.

One qualifier is easy to miss: the capability matrix is tiered, from Leading down to Supported, and the blog notes audio question answering was trained in English only. Forty languages does not mean forty at the same level; validate against the tier table before production.

The Scoreboard: 4.71% and 5.33% Both Stand

Two English WER figures stand simultaneously. Model-card basis: on 9 public test sets (collected October 12, 2026), English mean WER is 4.71%, next-best open model Qwen3-ASR-1.7B at 4.94%; the 7-language multilingual average is 4.62%, also lowest among open models, versus 6.49%. Leaderboard basis: on the Open ASR Leaderboard site, first among open weights at 5.33% (API models excluded). The gap is documented: as of October 9 the leaderboard began averaging in the private held-out sets of Appen and DataoceanAI plus URGENT24, while the card's 4.71% reflects the older 9-public-set basis. Cite either, label the basis, never mix them.

Per the leaderboard site as of October 9, 2026 (Nemotron at its release figure):

ModelLicenseAvg WERNote
Zoom Scribe v2 ProProprietary4.70Closed reference
Nemotron 3.5 Transcribe 30B-A4BOpenMDW-1.15.33First among open weights; 4.71 on the card's 9-set basis
Qwen3-ASR-1.7BApache-2.05.404.94 on the card basis
Cohere Transcribe 2BApache-2.05.77First among open on long audio (12.23%)
Canary-Qwen-2.5BCC-BY-4.05.94English only
Parakeet TDT 0.6B v3CC-BY-4.06.18RTFx 6,132
Voxtral Small 24BApache-2.06.38AA-WER 2.77%, 24B active
Whisper large-v3MIT7.1699 languages
Whisper large-v3-turboMIT7.64809M, no translation

Third-party numbers agree: on Artificial Analysis's AA-WER, Nemotron scores 2.81% (relayed by NVIDIA as 2.8%), tied with Voxtral Small's 2.77% atop the open field — while activating roughly 4B parameters against Voxtral's 24B, about a sixth per step. On the voice-agent slice AA-AgentTalk it posts 1.8% against 2.3% for the next-best open model.

Hard cases hold: long-audio mean WER of 10.98% versus 13.37% for the runner-up Parakeet-TDT-0.6B-v3; mean cpWER of 15.80 across the full-recording sets AMI, CALLHOME, Mixer 6, and NOTSOFAR-1, versus 16.83 for DiCoW v3.3 plus DiariZen. On style controllability, Preference-ASR shows 7.56 default and 6.16 instruction-following WER, against Canary-Qwen's 8.16 and 8.36.

Speaker attribution is the centerpiece, backed by two independent benchmarks. On David AI's DAI-ASR-I18N, spanning 21 languages and 14 systems, Nemotron Transcribe posts the lowest diarization error rate, 18.26%, narrowly beating NVIDIA's own dedicated Nemotron 3 Diarization at 18.38%; the combined transcription-plus-diarization track ranks first among open models, second overall. On VoiceArena (English): lowest cpWER among 14 systems, 32.4%, versus 34.2% for Multitalker Parakeet plus Nemotron 3 Diarization. More striking are deletions: 8.32% of reference words against a median near 31% — others drop whole sentences; it drops far less.

Throughput and Deployment: Spectacular Numbers, One Caveat

Official throughput first, qualifier attached: on the optimized vLLM inside the Nemotron Transcribe NIM, a single RTX PRO 6000 Blackwell Server Edition with NVFP4 and DFlash2 speculative decoding reaches up to 2,238 audio-seconds per second, returning a 30-second recording in 0.13 seconds; an alternative framing has 100 hours of audio processed in under 3 minutes (high concurrency on 30-second chunks). NVIDIA itself states the caveat: these service optimizations have not landed in the open-source release — self-hosted vLLM will not hit NIM numbers.

The deployment anchor is a single H100 80GB; the single-GPU range runs from Jetson AGX Thor, DGX Spark, and RTX Spark up to Blackwell data-center cards. Serving offers vLLM (NeMo Speech plugin) or SGLang-Omni, both with OpenAI-compatible chat endpoints. Inference optimization is native Multi-Token Prediction speculative decoding — the model's own MTP layers are the draft model — with a separately trained DFlash2 as an option.

Weights ship in four flavors: BF16 (full-precision reference), FP8, NVFP4 (for Blackwell), plus GGUF. GGUF needs its own paragraph: the research page already lists a GGUF companion repository aimed at NeMo-Speech.cpp for local inference — not llama.cpp, and not the community GGUFs around the Lightning line — while the official Hugging Face blog at launch still said quantized versions were "coming soon." Published GGUF weights do not mean a ready local toolchain; NeMo-Speech.cpp support is whatever the official repositories say, so do not write "one-click local inference" before it exists.

Hosted endpoints are available from Baseten, DeepInfra, and Simplismart, with demos on HF Spaces (nvidia/nemotron-transcribe) and ModelScope Studios. Artificial Analysis measured the DeepInfra endpoint at roughly 75 times realtime — one minute of audio in under a second — priced at $2.00 per thousand minutes.

One confusion to head off: NVIDIA also ships a sister model, nemotron-3.5-asr-streaming-0.6b — a 600M native streaming ASR with five latency tiers (80/160/320/560/1120 ms), 40 locales (32 out of the box), realtime on pure CPU. That is the streaming line; the 34B offline transcriber here is a different product.

Granary v2 and Fine-Tuning: Two Hour-Counts and a Medical Case

The training data hub is Granary v2, with two hour-counts that must be kept apart. The public manifest: 463,062 hours, 48 languages, 16 source corpora (Granary v1 covered only 25 languages). The full processing set: roughly 843,600 unique audio hours. Translation coverage adds about 142,800 hours translated into English and about 55,000 hours of English translated into 31 target languages. NVIDIA states these figures describe the dataset release, not Transcribe's complete training mix.

The dataset offers three transcription targets matching the model's promptable styles: tn_raw (text-normalized verbatim), itn_raw (inverse-normalized, fillers kept), itn_clean (filler-free). Its four-stage pipeline — Metadata, ASR, Text, HiFi — is built with NeMo Curator and supports Ray and NVIDIA Dynamo backends, with numbers behind it: the HiFi stage filtered VoxPopuli from 354,000 hours down to 111,000, and label-quality filtering moved the 9-benchmark average WER from 4.91% to 4.67%.

The model's own training scale: about 1.72 million unique indexed audio hours, 372 million samples, and 5.6 billion unique text tokens, mixing Granary v2, multi-speaker meeting corpora, audio-understanding data, and synthetic PII-aware speech — trained via supervised balanced mixing (sampling by answer-token quality) with one SFT stage plus a short RL phase. Worth crediting: some data was licensed from David AI, whose DAI-ASR-I18N evaluation set is explicitly excluded from training — the leaderboard win's independence is accounted for.

The fine-tuning bar is friendly: a single GPU adapts new languages, domains, or tasks, 17 tasks typically costing 0.5 to 2.3 GPU hours each; the NeMo Speech recipe trains the speech encoder and projection layers while the language model gets LoRA. Tooling lives on GitHub: NVIDIA-NeMo/Speech, repository: github.com/NVIDIA-NeMo/Speech — the SALM toolkit includes LoRA merging, a vLLM evaluator, and 16 tutorial notebooks, with the recipe hub in NVIDIA-NeMo/Nemotron (Apache-2.0) containing the Transcribe recipe map.

Healthcare example: on Eka Care's Indian medical voice benchmark (3,619 clips), prompt tuning alone — specifying English output in Latin script — cut the WER from 30.8% to 23.8%; after fine-tuning, the speaker-held-out 2,764 clips went from 22.44% to 12.17%, a 45.8% relative drop, with the whole process costing about 14 GPU hours (the official case ran on a single GB300); adaptation data was about 153 hours of synthetic speech, 29 hours of heavily accented Common Voice English, and 63 hours of MultiMed medical recordings. Prompt tuning costs zero training compute; fine-tuning stays far short of a major project.

FAQ

Q1: Can I use it commercially?

A1: Yes. The model card and research page both tag OpenMDW-1.1, explicitly covering commercial and non-commercial use; redistribution and fine-tuning are permitted, with no restrictions on model outputs. One obligation sits with the user: confirm the rights to a recording before transcribing it.

Q2: Can a consumer GPU run it?

A2: The reference deployment is a single H100 80GB, and the official single-GPU range spans Jetson AGX Thor, DGX Spark, and RTX Spark up to Blackwell data-center cards — DGX Spark and RTX Spark are desk-reachable. GGUF weights are listed, aimed at NeMo-Speech.cpp, but the local toolchain's maturity is whatever the official repos show. NIM throughput figures (2,238 audio-seconds per second) come from the optimized serving stack; self-built vLLM will not match them.

Q3: Can it do live captions?

A3: No. NVIDIA states this is a batch offline model without streaming support, and internal audio chunking is not a streaming service. For live captions, the sister model nemotron-3.5-asr-streaming-0.6b is a 600M native streaming ASR with latency tiers from 80 ms up, under the same OpenMDW-1.1 license.

Q4: Is the WER 4.71% or 5.33%?

A4: Both, on different bases. The 4.71% is the model card's English mean WER over 9 public test sets (collected October 12, 2026); the 5.33% is the Open ASR Leaderboard site's figure, which since October 9 has averaged in private held-out sets plus URGENT24. Cite either with its basis labeled — never mix them in a comparison.

Q5: How good is the speaker attribution, really?

A5: Independent benchmarks say strong: lowest diarization error rate among 14 systems on David AI's DAI-ASR-I18N across 21 languages, at 18.26%; on the English VoiceArena, a deletion rate of just 8.32% against a median near 31% — far fewer dropped words. But NVIDIA advises spot-checking names, speaker transitions, and audio QA results — keep human review in the loop.

Related reading

This article is AI-assisted and human-edited. Last updated: 2026-10-29

FAQ

Can I use it commercially?
A1: Yes. The model card and research page both tag OpenMDW-1.1, explicitly covering commercial and non-commercial use; redistribution and fine-tuning are permitted, with no restrictions on model outputs. One obligation sits with the user: confirm the rights to a recording before transcribing it.
Can a consumer GPU run it?
A2: The reference deployment is a single H100 80GB, and the official single-GPU range spans Jetson AGX Thor, DGX Spark, and RTX Spark up to Blackwell data-center cards — DGX Spark and RTX Spark are desk-reachable. GGUF weights are listed, aimed at NeMo-Speech.cpp, but the local toolchain's maturity is whatever the official repos show. NIM throughput figures (2,238 audio-seconds per second) come from the optimized serving stack; self-built vLLM will not match them.
Can it do live captions?
A3: No. NVIDIA states this is a batch offline model without streaming support, and internal audio chunking is not a streaming service. For live captions, the sister model nemotron-3.5-asr-streaming-0.6b is a 600M native streaming ASR with latency tiers from 80 ms up, under the same OpenMDW-1.1 license.
Is the WER 4.71% or 5.33%?
A4: Both, on different bases. The 4.71% is the model card's English mean WER over 9 public test sets (collected October 12, 2026); the 5.33% is the Open ASR Leaderboard site's figure, which since October 9 has averaged in private held-out sets plus URGENT24. Cite either with its basis labeled — never mix them in a comparison.
How good is the speaker attribution, really?
A5: Independent benchmarks say strong: lowest diarization error rate among 14 systems on David AI's DAI-ASR-I18N across 21 languages, at 18.26%; on the English VoiceArena, a deletion rate of just 8.32% against a median near 31% — far fewer dropped words. But NVIDIA advises spot-checking names, speaker transitions, and audio QA results — keep human review in the loop.

Related

Open Source

Stored 32 Layers, Run 56: SenseNova Fits 27B Into 9B

On October 30, 2026, SenseTime open-sourced SenseNova Loop-9B-Preview (github.com/OpenSenseNova/SenseNova-Loop, Apache-2.0): a Looped Transformer that stores only 32 distinct decoder layers, runs the middle 24 shared-weight layers twice, and executes 56 decoder-layer computations per forward pass - about 9B parameters with roughly 14.6B of compute, which the vendor reports matching 27B-class models. A two-stream mHC keeps information flowing across passes, with 256K context, multimodal input, and a torchtitan fine-tuning recipe.

Nov 3, 20269 min read
Open Source

DeepSeek Open-Sources Agent Orchestration: Let AI Manage AI

On October 31, 2026, DeepSeek Harness unveiled the architecture behind its Super-agent project and open-sourced the prototype (github.com/deepseek-ai/dsh-superagent-prototype, MIT, about 9,374 stars as of November 2). The core idea ports classic systems thinking - scheduling, caching, compute-storage separation, watchdogs - into an agent: a two-tier Router plus Worker, event-driven, async multi-channel, external Watchdog. In synthetic-load experiments half of tasks finished in 12.4 minutes and 95 percent in 15.7, but at the cost of more tokens - trading tokens for time and stability.

Nov 2, 20269 min read
Open Source

Breeze TTS 2: Open Weights No.1, Yet Not for Sale

BreezeBlue (Shanghai, founded June 2025, 15 people; founder Yang Bin is ex-MiniMax tech partner; 6 million USD seed announced August 25) open-sourced Breeze TTS 2: repo breezeblue-ai/breeze-tts (1,230 stars, Apache-2.0 code), about 3.48B parameters, Chinese-English bilingual, weights on Hugging Face and ModelScope. On Artificial Analysis's Provider Voice arena (late-August snapshot) it scored Elo 1215, first among open weights, 90 points over Fish Audio S2 Pro (1125); but on the controlled-voices board (8 fixed voices) it ranks only third among open weights (1002, tied with Fish S2 Pro, behind Mistral Voxtral TTS at 1010) and sixth overall — "first" needs its board name. README figures: under-40ms first audio and RTF 0.32 on a warmed H100 fast path, about 7.7GiB VRAM eager (12GB card advised) or 14.4GiB with --fast-all (24GB; the fast path costs more VRAM); four-stage architecture — T5Gemma2 text encoder, Qwen3-style 28-layer backbone, 12-layer depth decoder, Mimi-style VQ (16 codebooks, 12.5Hz). The hook is the dual license: code Apache-2.0, but weights, derivative models, and self-hosted outputs are research/non-commercial — commercial use needs written approval from RESONIA or the hosted platform (34 USD per million characters); community INT8 quantization runs it on an 8GB card (one developer measured a 7.4GB peak), plus a ComfyUI node pack and a community MLX 4bit port.

Oct 31, 202611 min read