While making my assistant talk, it never answered me
I gave my assistant BoBe a voice that runs on my own Mac, with a budget of 1.2 seconds to answer. The first evening it didn't answer at all, and every reason was about waiting.
When two people talk, the gap between one speaker finishing and the other starting is usually about a fifth of a second. Linguists have measured it across ten languages, and it’s remarkably constant. That’s why a voice assistant that takes two seconds to answer feels broken, even when the answer is good.
In May I set out to give BoBe a voice. BoBe is an AI assistant I’m building: a small face on the Mac’s desktop that remembers what matters to you and works on your goals. Until then you typed to it. I wanted to talk to it out loud, with the recognition and the speech running on my own Mac and no recording ever leaving it.
A budget of 1.2 seconds
Before writing code I wrote a budget. It runs from the moment you stop talking to the first sound of its answer, and it needs five steps. Something has to notice the sound of a voice. Something has to decide that you’ve finished. A recognizer turns your words into text, a language model writes a reply, and a speech engine starts saying it.
| Hear that someone is speaking | 10 ms per slice |
| Decide you've finished | 12 to 65 ms |
| Final text of your sentence | about 100 ms |
| Language model starts answering | 300 to 600 ms |
| First sound of the reply | 250 to 500 ms |
Added up, that’s 0.85 to 1.15 seconds, inside a ceiling of 1.2. The plan said plainly where it would break: the ceiling holds only if the language model starts answering within 600 ms. For the times it doesn’t, BoBe has short fillers ready. A quick “let me check” while a tool runs keeps the silence from stretching.
The hard part is knowing you’ve finished
Speed turned out to be the easy half. The hard half is deciding when you’ve stopped talking. If it waits for half a second of silence, it cuts you off every time you pause to think. If it waits two seconds, every reply feels slow.
The plan used a small model for that decision. Pipecat, an open-source toolkit for voice agents, publishes one called Smart Turn. It listens to the last eight seconds of audio. From the sound of your voice, and from more than silence, it judges whether you sound finished. It’s 8 MB and answers in a few tens of milliseconds on a laptop.
The evening it never answered
On 13 May I spoke to BoBe through the new pipeline, and nothing came back. Three separate problems were in the way, and I found them one after another that evening.
The first problem looked like a hang. BoBe’s background program sat at 100% of a CPU core and stopped answering even its health check. When a program is stuck like that, you can sample where it is thousands of times and see where it spends its time. Every single sample, 1,611 of 1,611, was in the same few lines.
Those lines asked two questions that sounded like one. The loop kept running while “is there speech?” was true, and inside it asked “give me the next finished stretch of speech”. While I was in the middle of a sentence, the first answer was yes and the second was nothing yet. The loop never moved on, and the program that should have fed it more audio was the program stuck in it.
The fix loops on the finished stretches alone, and the misleading question is gone.
The second problem was a timeout. Every request to the background program had 30 seconds to finish, a sensible rule for ordinary web requests. It also applied to the live voice connection, which is supposed to stay open for the whole conversation. Anything longer than 30 seconds was cut off, and the microphone button got stuck halfway.
Waiting politely, forever
The third problem needed a better test than my own voice. I had BoBe’s text-to-speech say a sentence and fed that recording into its own ears. It went through the real connection, in the same small audio packets the Mac sends. The words arrived one by one as text, and then nothing happened.
The gate I had put on the turn model wanted a confidence of 0.7 before it believed I’d finished. For complete, ordinary sentences, Smart Turn said about 0.55. BoBe was politely waiting for me to finish, and it would have waited forever.
Pipecat’s own reference uses 0.5. With that, the same test went all the way through. A final transcript came, then a short reply, then 9.5 seconds of its voice. It also explained why I’d heard nothing earlier in the day. Still, 0.55 against 0.5 is a thin margin.
Interrupting BoBe, and “uh-huh”
A real conversation also lets you cut in. If BoBe is talking and you start speaking, it should stop. But people say “uh-huh” and “yeah” while they listen, and those shouldn’t stop it. The Mac notices that you’re speaking from how loud the sound is, which can’t tell “uh-huh” from “wait, stop”. So BoBe only stops once the live transcript holds three words, the same rule Pipecat uses.
Where it ended up
Two days later the Mac took over the listening, so only words travel to BoBe’s background program, never audio. In October the Mac app switched to Apple’s own speech recognizer. The app got 23% smaller, and nobody has to download a 600 MB speech model before BoBe can hear them. The price is that the live text of your words updates about once a second, where it used to update three times.
The clever turn model is gone too. Today BoBe ends your turn after a pause you choose. It’s 400 ms if you want it quick, 800 by default, or a second and a half if you like to think out loud. After all that modeling, BoBe simply asks you how patient it should be.