I built a voice avatar for this site. A recruiter speaks, it answers as me. The first version ran on RunPod serverless GPU — Faster-Whisper for speech-to-text, a quantized Mistral 7B through Ollama for reasoning, F5-TTS to speak in a clone of my voice, Qdrant for retrieval. About ₹520–620 a month.
I never switched it on. It sounded robotic and useless.
Then I went looking for why, expecting to find a model problem. I found five causes and only one of them was the model.
What was actually wrong
The persona file was fed to the model raw. mini_me_persona.md doubled as a
template for its own author, so the system prompt literally contained
INSTRUCTIONS FOR SWARNA: fill in every section below and nine
[SWARNA: Replace the above with a 2-3 sentence bio] blocks. 2,154 characters
of scaffolding went into every prompt. A small quantized model handed
contradictory meta-instructions produces exactly the confused, generic output
that reads as "the AI is bad".
It forgot its own conversation. History was truncated to the last six turns, with a comment explaining this was to "stay within Mistral's context window". Mistral was long gone by then. A full ten-minute interview is around 2,400 tokens of dialogue; with the persona and retrieved context the prompt lands near 4,200 — against a 131,072-token window. It was throwing away the start of its own conversation to save 3% of a window it was nowhere near filling. The fix was deleting a slice, not building a summariser.
It misgendered its own subject in nine places, including the hardcoded fallback opening that was spoken at the start of every session.
Cold start was structural. RunPod's own FlashBoot case study reports a p95 of 2.3s and a 42-second maximum, and says plainly that staying warm needs "a good volume of requests". At roughly five sessions a day there is nothing to keep warm. The endpoint's job history told the same story: 43 completed, 14 failed — a 25% failure rate.
F5-TTS could not stream. Its published 253ms is a complete forward pass — audio cannot begin until synthesis finishes. What a listener experiences is time-to-first-byte, so a model with a worse real-time factor but genuine streaming feels faster. I had optimised the wrong number.
The licensing trap
F5-TTS: "Our code is released under MIT License. The pre-trained models are licensed under the CC-BY-NC license due to the training data Emilia."
Same split in XTTS-v2 and Fish Speech S1-mini. Whether a personal site counts as "commercial" is arguable — and building a public service on an arguable reading of a non-commercial licence is an avoidable risk when Apache-2.0 alternatives exist (Kokoro-82M, CosyVoice, Kyutai TTS). NVIDIA's original Canary-1B is CC-BY-NC-4.0 too; the v2 variants relicensed.
The permissive licence on the code is not the licence on the weights. I had read the first line and stopped.
What replaced it
The browser talks to Gemini Live directly. My server mints a single-use ephemeral token and never touches the audio.
| RunPod | Gemini Live | |
|---|---|---|
| Idle cost | ₹520–620/mo regardless | ₹0 |
| Per 8-min session | (already paid) | ~₹8.60 |
| Cold start | 10–42s | none |
| Turn latency | multi-second | p50 808ms / p95 917ms |
| Licence risk | CC-BY-NC weights | none |
The GPU was the entire cost of the feature, and it turned out to be optional.
What I gave up is the voice clone. On a hiring page I think that was the right trade — a stranger's voice reading my answers was a weaker idea than it sounded.
Four bugs that all looked like "the AI is bad"
Died after exactly one turn. session.receive() yields messages for one
turn and then the generator ends. A single async for handled one exchange and
returned, so nothing drained the websocket, keepalive pings went unanswered, and
the socket closed with 1011. It looked like a model failure. It was me
misreading an API contract.
Froze while speaking. speaker.write() blocks until the audio device
drains, and I was calling it from inside the receive loop — so the event loop
was frozen for the entire time the avatar spoke, and the microphone never
pumped.
It interviewed its own echo. Every "YOU" line in the transcript was the
avatar's own previous sentence. Raw PCM capture has no echo cancellation. The
browser does this for free: echoCancellation, noiseSuppression and
autoGainControl are one line of getUserMedia constraints. My desktop test
harness was the worst possible environment for this workload, and the browser —
where the product actually lives — was the best.
A latency number that measured the wrong thing. I was timing "response gap" from first-audio-out to turn-complete, which is how long the avatar spoke. A good four-sentence answer reported "12.76s" and read like a catastrophe.
The one that mattered most
The first genuinely good session — fourteen coherent turns, sub-second — ended with a visitor asking for contact details.
It read out my personal mobile number and email address, aloud, immediately.
resume.json was being dumped wholesale into the model context, personal.phone
included. On a public site that is a scraping target with no rate limit in front
of it.
A voice interface turns every field in your data into something that can be read aloud to a stranger who asks nicely.
That is the sentence this whole exercise bought.
Smaller things
- Groq sits behind Cloudflare, which 403s the default
Python-urllibUser-Agent witherror code: 1010. This looks exactly like a revoked key. I regenerated one for nothing before reading the response body. gemini-2.0-flashfree-tier quota is nowlimit: 0— the fallback leg of a two-model cascade had been silently dead.- ASR mangles some names regardless of accent. "Swarnakeshar" came back as "Shounak Kishor", "Swarnakeshwer", "Swarnakesh" — including on clean synthetic audio. The fix is not better pronunciation. It is using the short form in speech.
- No primary source publishes India-accent WER, so I measured it: 94.3%
cross-model agreement on real recorded speech, with Gemini beating Groq's
Whisper on every disagreement, particularly on technical vocabulary
(
pgvector→ "pg vector",groq→ "grok",faiss→ "fass").
What I'd take to the next one
Measure the thing the user experiences, not the thing the library reports. Time-to-first-byte, not forward-pass duration. And read the weights licence, not the repository licence.