A latency budget for voice agents that look things up mid-call

Why lookups break latency

A voice agent has roughly 800 ms before a caller notices a gap. Speech recognition, the model and speech synthesis already take most of it. A web lookup in the middle of a turn can add another 600 ms on its own.

This post splits that budget stage by stage and shows where the time goes on a real call.

The budget

Our reference agent spends 140 ms on transcription, 260 ms to the first model token, 90 ms on the lookup when it hits a regional cache and 180 ms to the first audio frame. That leaves about 130 ms of slack.

The lookup only fits if it runs in the same region as the model and starts before the model asks for it.

Starting the lookup early

We stream partial transcripts to a small classifier. When it predicts a lookup with more than 70% confidence, the query starts while the caller is still talking. Wrong guesses cost one request and are dropped.