Getting my coding agent to answer by voice
I wanted my coding agent's answers read back to me with nothing extra to install. The library I planned on broke one of my program's own rules.
In April I added voice to tebis, a small Rust program I wrote that lets me drive the coding agents on my computer from a Telegram chat on my phone. Voice notes going in were the easy half. What I wanted next was the agent’s answer read back to me as a voice message, on a Mac or on Linux, with nothing extra to install.
The Rust library I planned to use for that broke one of tebis’s own rules, and none of the three other Rust libraries for the same voice model worked either. So I ported a speech pipeline to Rust myself, and on Linux the install I had tried to avoid came back anyway.
From a voice note to a spoken answer
tebis runs on the computer where the agents work. Each agent runs in tmux, a program that keeps terminal sessions running in the background and lets other programs type into them. A message I send to my private Telegram bot gets typed into the agent’s session, and the agent’s answer comes back to the chat.
Voice notes going in took one evening. Telegram sends a voice note as a finished audio file, and tebis turns it into text with Whisper, OpenAI’s open speech recognition model, which runs inside tebis. Then it types the text in like any other message.
The way back needed a voice. I picked Kokoro, a small open model that turns text into speech. Kokoro doesn’t read letters. It reads phonemes, symbols for the sounds in a word, and many ways of running it get those from espeak-ng, a separate speech program you install yourself. That was the install I wanted to spare people.
The library I planned on broke a rule
My plan picked a Rust library called any-tts because it worked out the phonemes itself, inside the program, so nobody would need espeak-ng. But it pulled in a library that tebis doesn’t allow.
tebis can type into my shell and holds the token for my Telegram bot, so it is strict about the code that talks to the network. It reaches Telegram through its own small web client, and all of its encryption goes through a single library. A build check fails if anything pulls in a library from a short ban list. Every extra library there is more security code to trust. any-tts pulled in reqwest, a popular Rust library for web requests, and reqwest is on that list.
So that night the first voice replies came from the Mac’s built-in speech command, and Linux had no voice at all.
None of the four libraries worked
Linux still needed a voice, so after midnight I wrote a second plan. It kept any-tts but switched off its model downloader, the part that pulled in reqwest, since tebis already downloads its own models.
Twenty-five minutes later I marked it blocked. Another part of any-tts still pulled reqwest in, and with it a second encryption library. By then I had tried all four Rust libraries for Kokoro, and none of them worked inside tebis. Two didn’t compile at all, and the other two only built by breaking one of its rules.
There were ways around it. I could drop the rule, keep my own copy of any-tts with the web library cut out, or run Kokoro through Python and make Python the extra install. I turned them all down, and the commit left a note to consider writing my own if nothing changed in three months.
Porting the pipeline myself
I didn’t wait three months. The next afternoon I ported to Rust the pipeline of Kokoro-FastAPI, an open-source Python server that runs Kokoro, so tebis could speak on Linux with nothing running beside it.
The port gets its phonemes from espeak-ng, so the install any-tts was meant to spare me was back. Most of the work went into the text around that step. My first version passed espeak-ng’s phonemes straight to Kokoro and sounded robotic, because Kokoro learned from phonemes spelled a little differently. The port now spells out numbers, years and prices, and rewrites the phonemes into the spelling Kokoro expects.
It came to 1,615 lines of Rust, about a third of them tests, in a program that had been about 15,000 lines. With any-tts doing that work, the second plan had budgeted about 250.
Two installs came back
So on Linux, tebis now needs espeak-ng. It needs a second install too: ONNX Runtime, the library that runs the Kokoro model. Building it into tebis would have added about 20 MB to a program of about 5 MB, so tebis loads it from the system while it runs.
The setup looks for both and offers to install whatever is missing. Fedora and Alpine have no package for ONNX Runtime, so there the setup prints the steps and you do them by hand.
| Measure | Planned, with any-tts | My own port |
|---|---|---|
| Lines of code for the voice | 250 | 1,615 |
| Programs to install on Linux | 0 | 2 |
What I learned
A library’s description doesn’t tell you what it pulls in. Both plans picked any-tts for what it promised. Building it took minutes and showed the banned libraries behind it.
Most of a speech pipeline is preparing the text. More than half of the 1,615 lines spell out numbers, get phonemes from espeak-ng and fix them. Without that, the model sounded robotic.
Keeping the rule cost a port and two installs. I could have relaxed it in a few lines of config. I kept it, and paid with my own pipeline and two installs on Linux, one of them the install I set out to avoid.
Where it stands
On a Mac, tebis answers with the Mac’s own voice by default and needs nothing extra. On Linux it answers with Kokoro once espeak-ng and ONNX Runtime are installed, so the install I set out to avoid is part of the setup there.
Either way, voice replies stay off until you turn them on, and tebis answers by voice only after you send a voice note. On Fedora and Alpine, ONNX Runtime still has to be set up by hand.