3 August 2026
I spent a month making a live assistant fast enough to be useful in a conversation. The first thing I had to throw away was the metric everyone reaches for.
The tool is a prompter for calls on a Mac. It listens to both sides, works out that the other person has finished asking something, and puts the beginning of an answer on screen. The whole thing is worthless if it is slow, because it is competing with a specific window: the pause you naturally take after a question.
That pause is about three seconds. You can stretch it with a filler sentence, and nobody notices. Past five seconds they notice. So the budget was fixed before any code existed, and it is an end to end budget: from the moment the other person stops talking to the moment I can start.
My first metric was time to first token, because that is what everything reports. It took one measurement to see the problem. I logged the actual content of the first streamed chunk across a run and looked at it:
The median first chunk was two characters. Things like Th, No, I. Twelve readable words arrived on the third chunk. A finished sentence arrived on the fifth.
So time to first token was measuring the moment at which there is nothing to read. Optimising it would have been optimising a number that no user experiences. Worse, it made the system look about 0.7 seconds faster than it was, which is a quarter of the entire budget.
I replaced it with time to twelve words. Twelve is not a round number I liked, it is roughly what a person needs on screen to start speaking without reading ahead of themselves. I will come back to how I checked that the threshold itself was not lying to me, because it very nearly did.
Measured in the shipping application, on live speech, on one Apple silicon Mac. Median across 23 turns, p90 3446 ms, slowest turn 4517 ms.
| stage | median, ms | whose |
|---|---|---|
| Speech to finished transcript | 418 | mine |
| Model, to first streamed chunk | 1199 | not mine |
| Streaming, until twelve words are on screen | 778 | not mine |
| Total, to twelve usable words | 2636 |
Two thirds of the budget belongs to somebody else. That is worth knowing early, because it sets a ceiling on what local optimisation can buy you, and it means the slow tail is mostly not yours to fix. My two worst turns were 4517 and 4124 ms, and in both the model took over 3.2 seconds to produce anything while my own stages ran at their normal speed.
Speech recognition runs locally, which is a privacy decision first and a latency decision second, but it turns out to be both.
Whisper large-v3-turbo did not fit at any setting. With a full window it costs about 972 ms of decoding, which pushed the end to end number to 3.3 seconds against a 3 second goal. Trimming the window makes it fast and makes it lose long questions, which is the failure you least want: you do not find out that the question was truncated, you just answer the wrong thing. The obvious hybrid, short utterances on a trimmed model and long ones on a full model, does not work either, because you only learn how long an utterance was after it ended.
Parakeet TDT 0.6b v3 decodes the actual length of the audio rather than a fixed window. On a warm process it costs 67, 141 and 317 ms for utterances of 2, 7.5 and 17.6 seconds, a real time factor between 0.018 and 0.034. The speed matters less than the shape: with a tail that small, you can start decoding speculatively before the person has finished, throw the result away if they keep talking, and an entire class of failure disappears along with the fixed window.
The standard approach is a silence threshold: when the voice activity detector has seen N milliseconds of quiet, decide the turn is over. I could not make it work, and I do not think it is fixable by tuning.
The pauses inside a live question overlap with the pause after it. At a 700 ms threshold, 9 of 21 real questions were cut in half. A threshold long enough to never cut anyone off cost about 1.9 seconds, which is most of the budget spent on waiting for silence that already ended.
So the decision moved to the text. The transcript is examined and asked a much easier question than "is this a question": is this a finished utterance. The silence threshold still exists, but it was demoted to something that merely presents the current transcript for judgement, and it dropped to 400 ms.
What made the design tractable was writing down the cost of each error before writing any rules. A needless trigger costs one wasted request. A missed ending costs a timeout, about 1478 ms instead of about 513 ms, a full second straight into the metric. The errors are not symmetric, so the rule set is not a classifier trying to be right, it is a filter that passes by default and only holds back obvious noise.
Those two numbers, 513 and 1478, come from measuring this stage on its own against a 63 sample corpus. The 418 ms in the table above is the same stage measured inside the running application on a different and smaller sample. They are not interchangeable and I try not to put them in the same sentence without saying so.
This is the part I would skip if I were writing an advertisement.
The first version of those rules was derived from seven transcripts and then evaluated on the same seven transcripts, which of course reported that it worked. Running it against a fresh corpus of 100 finished questions gave the real number: 18 missed endings out of 97. Roughly every fifth question would have fallen through to a timeout.
After three fixes, it was 3 out of 47 on a held out half that I opened exactly once. What the fixes taught me was more useful than the score:
Once the pipeline was fast, I asked the obvious follow up: should the threshold be higher than twelve words, so that the suggestion carries a complete thought? The data to answer it already existed, because every streamed chunk was logged with a timestamp, so the moment "N words are on screen" can be reconstructed exactly. Same 26 turns.
| threshold | turns that reach it | median, ms | vs 12 words |
|---|---|---|---|
| 12 words | 23 / 26 | 2636 | +0 |
| 15 words | 21 / 26 | 2756 | +120 |
| 18 words | 14 / 26 | 3269 | +633 |
| 20 words | 9 / 26 | 3024 | +388 |
| 22 words | 6 / 26 | 3463 | +827 |
| 25 words | 1 / 26 | 4972 | |
| 30 words | 0 / 26 |
Look at the second column, not the third. At 20 words the median is 3024 ms, which is lower than the 3269 ms at 18 words, and for about ten minutes I had a number that argued for a higher threshold. It argues for nothing. The median answer is 18 words long, so the 18 word threshold is undefined for half the sample, and the 20 word row is computed from the 9 longest answers, which came from questions the model answered fluently. The slow short turns simply left the statistic.
A measurement that quietly loses part of its sample returns a number better than the truth, and it looks exactly like a result. This is the same failure as tuning rules on your test set, wearing a different hat.
Collected here because each one produced a plausible number rather than an error, which is the only reason they were dangerous.
\b for a word
boundary, which does not fire on Cyrillic in JavaScript, so it had matched nothing
at all. The real number was 77 percent. A broken instrument returns a beautiful
number, not an exception.
I use speakers rather than headphones, so my microphone hears the other side of the call. Separating the two by text similarity works until the moment I repeat the question back, which is both a normal thing to do and indistinguishable from an echo by that method. Current leak is around one failure per 150 utterances in each direction. The real fix is acoustic echo cancellation on the waveform, which is parked with a known price rather than pretending it is solved.
The recogniser also does not put final punctuation on very short utterances, so "For example" and "And then" look identical to noise to the rule that requires a terminal mark. That one is a limitation of the model, not of the rules.
Not any of the numbers. This: decide what the user actually experiences, then check that your metric fires at that moment and not before. Time to first token fires when there is nothing on screen. A word threshold above the median answer length fires only for the answers that were going to be good anyway. A timer with no check on the text fires when the system has produced something useless. Every one of those told me a story I wanted to hear.