Five speech engines in six months

BoBe is a small AI companion that lives on your Mac. It talks, so it has to hear you and answer out loud, quickly, on your own hardware. Since May it has run on five different speech setups. Every change after the first one moved toward what the operating system already does, and every one deleted code.

The plan in May

The first plan, docs/voice-plan.md, came out of a round of research. It picked open models that run anywhere: sherpa-onnx with Moonshine for speech to text, Kokoro for the voice, Silero and smart-turn v3.1 to tell when you have finished talking, and a custom wake word. It turned down Apple’s SpeechAnalyzer and FluidAudio because they only run on a Mac, and BoBe also runs headless on Linux and in a small ESP32 body.

The budget was 850 to 1,150 ms from the end of your sentence to the first sound of the answer. The plan says that ceiling “holds only when LLM TTFT ≤600ms”, the time the model takes to send its first token. Most of the budget belonged to the model.

The changes

The next day Moonshine gave way to streaming Zipformer, so recognition runs while you speak. When the model is slow to start, a short filler (“Hmm, let me think.”) plays after 800 ms.

Two days later the end-of-turn threshold went from 0.7 to 0.5.

On 15 May the Mac app took over recognition with FluidAudio’s Parakeet. That model is about 600 MB, and the Mandarin one about 1.75 GB. The daemon kept only the text side: words in, the model, Kokoro out, about 340 MB.

In October a native Pocket TTS (about 169 MB, no Python) replaced Kokoro and eSpeak. On one local sample it made 6.56 s of audio in 0.43 s, with the first chunk after 47 ms.

Then I asked whether the macOS recognizer would do, thinking FluidAudio was already gone. It wasn’t, and on 20 October it went. The Mac now uses Apple’s SpeechTranscriber, and DictationTranscriber for Greek.

What Apple’s modules don’t tell you

They don’t agree on what they support. SpeechTranscriber covers 30 locales and no Greek; DictationTranscriber covers 54. supportedLocale(equivalentTo:) still returns el_GR for the module that can’t do Greek, so BoBe checks the module itself.

SpeechDetector, the part that should notice when someone stops talking, reported nothing, even across a 3.5 s gap. BoBe measures pauses itself now, at 400, 800 and 1,500 ms, and finalizes the text in about 30 to 45 ms.

With one engine per platform, the settings lost a four-way engine choice and 24 strings in every language.

Where the open models stayed

They stayed where there is no Mac: the headless daemon and the ESP32 body. Licences decided some of it. Voices under noncommercial licences were left out, a GPL lexicon was removed, and the plan notes that the default wake-word models are noncommercial too.

The current setup is in docs/speech.md.