Grai · Study
And why the figures vendors publish can't be compared with each other — not even when they carry the same name.
A vendor announces "under 300 ms". In the same article it also gives its own measurement: 0.9–1.4 seconds. That is not a contradiction. They are two different quantities with the same name.
"Under 300 ms" measures model inference on ideal hardware — from the last sample into the network to the first one out. The other figure starts the stopwatch at the server, sometimes with no network, almost never with the playback buffer included. Neither is wrong. They just don't measure the same thing, and the difference is larger than the entire gap between vendors.
Hume, a voice AI company, published the analysis "Measuring benchmark optimization in speech recognition" on 21 August 2026, which reaches the same conclusion for speech recognition: models are not only getting better, they are also getting optimized for the tests themselves. By their analysis, 6 of 11 open models reproduce the errors in the VoxPopuli reference transcripts, even when the audio says something else — meaning they have seen the test, not merely learned the task.
One of their detection methods is elegant: cut the numbers out of the audio and check whether the model writes them anyway. On LibriSpeech, some models do so in 30–40% of cases. If it does, it didn't hear — it remembered. These figures are declared by Hume, in English; we have not reproduced them.
The first five are mistakes we made ourselves, in our own measurements, and caught later; the sixth is one we are exposed to right now. That is why they are written here with examples, not as an abstract list.
We published "455 ms to answer" internally. That figure started the stopwatch after speech recognition. Measured at the client, from the last sound the person makes — the only point the person actually feels — the real figure was 1235 ms to first sound and 2667 ms to the first sound of the answer (p50, n=29, filler on). To the answer, almost six times longer.
The question that breaks any press release: does it include the end-of-turn decision? In a second run, with the filler off (p50, n=17), that decision alone is 586 ms out of 3172 ms to first sound — more than the entire figure we had published. The two runs answer different questions and do not add up.
Many systems emit an "mm-hm" or a filler phrase that covers the wait. If the stopwatch stops at the first sound, the figure measures how fast the system makes a noise, not how fast it answers. The difference can exceed a second.
The de facto standard on public datasets spells numbers out in words before comparing. A model that hears "two thousand twenty-six" instead of "2026" is not penalized.
For us, the difference is measurable: utterances with digits score 29.65% normalized and 36.94% raw — 7.3 points hidden by normalization. For an agent that takes times, prices and phone numbers, that is exactly the part that matters. That is why we publish both columns.
Two vendor pages publish figures for the same two third-party models and they don't match: 12.31% versus 13.0% for one, 33.89% versus 35.4% for the other. Between two published "FLEURS Romanian" figures there are 0.7–1.5 points of difference from split and normalization alone. A paper (arXiv 2509.14128, Tab. 11) even gives Whisper large-v3 8.24%.
The practical consequence: any gap under two points between two vendors means nothing without a run on the same material.
A model that gets the whole file, offline, with 1.5 billion parameters, in a data center, is not solving the same problem as one that emits partial text while the person is still speaking, on an office Mac, with 0.6 billion.
We made this mistake on our own site: we put the two figures side by side without their measurement conditions. The comparison came out against us, and it was just as invalid as one that would have come out in our favor. Putting one language in another language's ranking belongs here too: Russian does not sit in the FLEURS Romanian table.
A model that saw the test set during training scores better on that set than on new audio. Our speech recognition model, Nemotron, lists FLEURS among its training data, and our headline figure, 26.15%, is also on FLEURS. We don't know whether it also saw the test split; the figure may be optimistic. Our answer is a memorization probe on Romanian: we silence the numbers in the audio and look for reference errors the model reproduces. Until it reports, FLEURS stays a bridge for comparison with published figures, not the judge; the judge will be a Romanian set recorded by us, one the model has never seen.
Existing benchmarks measure speech recognition and, less often, latency. Almost nothing that makes a voice agent usable has a public benchmark.
How often it cuts the person off mid-sentence, and how long it makes them wait for nothing. Two opposite errors that can't be optimized at the same time — so one number says nothing without the other.
How long stopping takes, and — far more important — what remains in the model's memory afterwards. Nobody publishes this.
What the system does when it fails to understand twice in a row. A WER benchmark can't see the difference between a system that asks you to rephrase and one that makes things up.
Times, prices, phone numbers, dates. They are under 20% of the words and over half of the practical value of a business conversation.
When two people speak at once, on purpose or not. Every public dataset is monologue.
Of the existing conversation benchmarks, none includes Romanian. The figures we read about Romanian all come from speech recognition datasets of read-aloud utterances.
We publish every measurement with its sample, streaming window, confidence interval and reproduction command. We mark what we measured and what others declared. We put the conditions next to every figure. And we publish the figures that make us look bad — because a benchmark missing its bad results is not a benchmark, it's a brochure.
Every measurement, with sample, streaming window and confidence interval — including the ones where we lose.