Benchmarks and figures

Speech recognition

FLEURS Romanian and Russian

Model: Nemotron 3.5 ASR streaming, 0.6 billion parameters. Sample: 200 utterances per language from the test set, uniformly random, seed 20260831. Decoding on the production path — cache-aware, streaming — not offline decoding of the whole file. Hardware: M3 Max Mac, speech recognition through MLX on the integrated GPU, no dedicated graphics card.

Text normalization fixed on 5 September 2026 (thousands separators, times of day): 26.31% → 26.15% for Romanian, 9.30% → 9.15% for Russian. Same transcripts, rescored.

Four independent streaming windows, each with the NVIDIA model-card figure inside our confidence interval. Our figure is not a broken test — it is the model's known behavior, consistent with its maker's card. At 80 ms the model barely keeps up with real time (1.51×), so two simultaneous sessions would no longer fit on the machine. The model saw little Romanian in training: NVIDIA's model card places Romanian in its «broad-coverage» tier, the second of three, and lists FLEURS among its training data. Our figure agrees with the card, but may be optimistic; we are checking it with a memorization probe.

Language auto-detection costs +1.6 points (27.71% vs 26.15%; paired difference on the same utterances: +1.58 points, 95% CI +0.77 to +2.49). That is why we keep the language fixed wherever it can be inferred before the first turn; auto-detection stays a fallback.

Russian, same pipeline, same run

Romanian is 2.9× worse than Russian in the same model, on the same hardware, in the same run. It is not an impression lifted from someone's model card. Our bilingual agent does not have two equal languages; it has a good one and a weak one. The same memorization caveat applies to Russian.

Market comparison

With the measurement conditions alongside

The "how" column is what makes the table readable. A model that gets the whole file, offline, in a data center, is not solving the same problem as one that emits text while the person is still speaking, on an office Mac.

The offline ceiling, local, on the same 200 utterances

Measured by us, with the same normalization as Grai. Offline: the model gets the whole utterance and does not emit text while the person speaks — so it cannot stand in for streaming speech recognition.

What beats us on Romanian, on our own machine, is an open Romanian model, offline. The ~19-point gap is the price of emitting text while streaming; that is why the next step is distilling it into a small model that streams, not tuning Nemotron.

What we have instead, and they don't

• It runs locally, without the caller's voice ever leaving the machine.
• Streaming, with partial text while the person speaks — not after they finish.
• Free to run, no per-minute cost.
• On office hardware, not on a node with a dedicated accelerator.

Only one vendor on the list also offers on-prem with streaming Romanian — so it is the only one that attacks our argument head-on. Against it, the argument remains cost and control, not accuracy.

Where it breaks

Numbers. Exactly the part that matters on the phone.

Splitting the 200 utterances by whether the reference contains digits:

An utterance with digits is 4.6 points worse normalized and 11.5 points worse raw. Real examples from the run (in Romanian, as transcribed):

Any normalization that spells numbers out in words before scoring makes us look better than we are on the phone. That is why we also report raw scores. For an agent that takes phone numbers, times, prices and dates, the most important part of the language is also the most broken one. Why normalization matters →

Latency

From your last sound to its first

Two different runs, answering two different questions. They do not add up.

Filler on — what the person hears

n=29 turns, p50, measured at the client, from the person's last speech sample.

Filler off — where the milliseconds go

n=17 turns, p50, gpt-5.4-nano. Each segment has two observed endpoints, not computed ones — the six segments add up to the total within half a millisecond.

The 92 ms of speech recognition is the flush after the person stops talking; partial text during speech arrives with the streaming window's delay, ~1.1 s at 1120 ms. The only piece that does not run locally is also 69% of the delay.

Reference point: HumDial, ICASSP 2026, reports 1.26 s for the winner, under the challenge's own definition. It is a different quantity from the 1235 ms or 2667 ms above; one is not subtracted from the other. What the reference shows: where full-duplex is actually measured, the figures are on the order of a second, not 300 ms.

Turn detection

Prosody versus the stopwatch

Measured on real human conversations from CANDOR, in English: 80 clips, 98 pauses, offline replay. The metric is TOR — the share of mid-sentence pauses in which the system wrongly decided the person had finished. With a fixed 0.6 s threshold, 69.4%; on intonation, at a 560 ms wait, 52.0% — which the stopwatch needs 820 ms to reach. The gain does not come from deciding less often: the prosody decision makes more decisions than the fixed threshold, and still errs less. Public reference points: 9.9% false cuts at 300 ms and 4.5% at 600 ms, for LiveKit v1 — declared, English, a different metric (false-cut rate, not TOR), so not directly comparable.