--- title: AI Voice Agents on Asterisk: The Latency Budget category: Voice AI published: 2026-08-18 reading_time: 11 min canonical: https://technologiesninja.com/blog-ai-voice-agent-asterisk.html --- # AI Voice Agents on Asterisk: The Latency Budget Every architectural decision in a voice agent is a negotiation with the same 800 milliseconds. A text agent can think for four seconds and nobody minds. On a phone line, a pause past roughly 800ms reads as the call having dropped, and callers start saying "hello?" — which arrives as new input and makes everything worse. The entire design follows from that one constraint. ## Where the milliseconds go ``` speech-to-text 100 - 300 ms LLM inference 350 - 1000 ms text-to-speech 90 - 200 ms network hops 50 - 200 ms (per vendor boundary) ------------------------------------------------------- p50 target < 250 ms optimised stack p50 acceptable < 400 ms standard cloud services p95 ceiling < 800 ms ``` The LLM is the largest and most variable term, and the network line is the one people forget. Every vendor boundary is a round trip, so a pipeline stitched together from three providers in three regions can spend more time in transit than in inference. ## Streaming everything is not optional The naive pipeline waits for each stage to finish: transcribe the whole utterance, send it, wait for the full completion, synthesise, play. That is the sum of every stage, and it will not hold a conversation. - stream partial transcripts and start the model on a stable prefix - stream tokens into TTS and begin audio at the first sentence boundary - keep connections warm; TLS handshakes inside the budget are pure waste - co-locate STT, LLM and TTS in one region, even at the cost of a preferred vendor Done properly, the caller hears the first syllable while the model is still generating the end of the sentence. That single change is usually worth more than any model swap. ## Turn-taking is the hard part Knowing when the caller has finished is harder than understanding what they said. Fixed silence thresholds fail in both directions: too short and you interrupt someone who is thinking, too long and every exchange feels sluggish. - semantic endpointing — decide on the content, not only on silence duration - barge-in — the caller speaking must stop playback immediately, mid-word - discard the audio you were playing rather than resuming it after an interruption - handle the "hello?" case explicitly, because it will happen during your worst latency Barge-in is where self-built stacks most often feel wrong. If stopping playback takes 300ms the agent talks over the caller, and no amount of model quality recovers from that. ## What Asterisk gives you Putting Asterisk or FreePBX in front of the AI stack is a deliberate choice rather than nostalgia. You already have SIP trunking, call recording, IVR, queues, transfer and failover — solved, and audited. - media access through ARI, or a raw audio channel for streaming - transfer to a human as a native operation rather than a vendor feature - recordings and call detail records staying on your own infrastructure, which matters for regulated work - per-minute cost that is your trunk rate rather than a platform's margin The cost is that you now own the media path. Packet loss, jitter and resynchronisation are real on the PSTN, and synthesised audio has to tolerate retransmits without audible artefacts. A managed platform hides that work; self-hosting means doing it. ## Design for the failure, not the demo Voice agents fail in public, in real time, to a customer. The escalation path matters more than the happy path. - transfer to a human on repeated low confidence, not only when asked - a hard turn limit, after which the call routes to a person - log audio alongside transcripts; text alone hides mishearing - watch the 95th percentile, never the average — the average call is fine, and the bad ones are what people remember The measure of a voice agent is not how well it handles the question it was built for. It is how quickly it gets out of the way when it cannot. --- ## Read next - Case study — Asterisk VoIP → /project-asterisk-voip - Article — Voice AI with Whisper, WhisperX and Ollama → /blog-whisper-voice-stack - Service — VoIP & Voice → /service-voip-voice --- © 2026 TechnologiesNinja · Built and maintained in-house