What Is Voice AI Observability? Signals, Metrics, and How It Works
Voice AI observability captures every production call and judges whether the agent did its job. Here's what it captures, what to measure, and how failures become tests.
A live voice agent gets two common checks: manual QA, where people listen to a sample of calls, and uptime and latency dashboards, which show whether the agent is up and fast but not what it said. Neither judges every call.
Voice AI observability does. It captures every call your voice agent handles in production and automatically judges whether the agent did its job, so you can spot the ones that went wrong and have a clear trail for escalations or auditors.
Capturing calls is the easy part. The value comes from evaluating the calls: scoring every one on the same metrics you test with before launch, checking the AI judge (the model that grades each call) against human reviewers, and turning each failed call into a regression test the next release has to pass. Here’s how both work, starting with what a call leaves behind.
What automated voice AI observability captures
Every call leaves four kinds of evidence behind. Captured together, they let you replay the call and find the stage that caused a problem.
- The conversation: what did the caller hear? Stereo audio and a transcript with turn timestamps. The recording keeps what logs miss, and the timestamps make every gap measurable. With the agent and caller on separate channels, Coval knows who said what from the channel itself.
- The trace: where did it break? Speech-to-text, the LLM that writes the reply, text-to-speech, and each tool call get their own span, a timed record of that step. Coval reads stt, llm, tts, and llm_tool_call spans over OpenTelemetry, so a bad call points to the words the agent misheard, the reply it generated, or the function it called with the wrong arguments.
- The context: which release did this? The prompt and model version on every call let you compare one release with the last.
- The outcome: did it work? Whether the request was resolved, handed to a person, or dropped when the caller hung up.
Getting this data in is the solved part. Coval’s tracing wizard auto-instruments a Pipecat, LiveKit, or Vapi agent with one command, and teams already tracing with Langfuse, Arize Phoenix, or LangSmith can import those traces without re-instrumenting.
How voice observability differs from manual QA and uptime dashboards, and why voice needs it
Having every call on record still doesn’t tell you which ones went wrong, and each of the two common checks leaves a gap. Manual QA judges what the agent said, but only on the calls someone has time to hear, so a rare failure can slip through unheard. Uptime and latency dashboards cover every call but judge none of them: they track whether the agent is up, fast, and error-free, not whether the caller got the right answer. Voice observability closes both gaps: automated scoring covers every production call, and it judges the conversation itself, not just whether the service responded.
Here’s the dashboard gap in practice. One of our customers thought its voice agent was transferring callers to its internal team. Someone else was on that line, a squatter (Brooke tells the story in our benchmarks AMA), so every transferred caller reached the wrong party. Every transfer still went through, so the uptime and latency dashboards had nothing to flag. They could only say the transfers connected. Whether a call reached the right party is a question about the call itself, and asking it of every call is what observability adds.
Google’s SRE book names four golden signals for a user-facing service: latency, traffic, errors, and saturation. Its own definition of an error already covers the voice problem. An error can be implicit, such as “an HTTP 200 success response, but coupled with the wrong content.” A voice agent’s wrong answer is exactly that. Speech-to-text returned a transcript, the LLM returned tokens, text-to-speech returned audio, and the caller heard the wrong refund policy. Uptime and latency dashboards count only the errors a service reports, and on that call every service reported success.
Voice makes behavior harder to judge than a chat log. A call runs many turns, and each turn can fail on timing as well as words: a reply that arrives late, an agent that talks over the caller, or a sentence lost to line noise. Tool calls happen mid-call, out of the caller’s hearing, and a wrong argument can sit behind a perfectly polite sentence. None of that shows up in the words alone. That’s why we say voice agent evaluation has to judge the whole call, caller emotion included, not just the final transcript.
Behavior also changes when nobody ships. Stanford and UC Berkeley researchers found GPT-4’s accuracy at identifying prime numbers fell from 84% to 51% between its March and June 2023 versions, under the same model name. An agent built on a hosted model can change without a deploy on your side, and only scoring its calls will show it.
What to measure with voice agent observability
So what do you score? Most of it falls under four questions about the call, and in Coval each one maps to a metric type:
| What you measure | Example | Scored by |
|---|---|---|
| Did the call succeed? | Request resolved with the correct policy information | An AI judge reading the transcript or listening to the audio |
| How did it sound? | Latency, Interruption Rate, and Background Noise | Statistical metrics computed from the recording |
| Where did it break? | Speech-to-text word error rate, time to first byte per stage, and tool call count | Trace metrics from OpenTelemetry spans |
| Were the hard rules followed? | A required disclosure line, a field in the right format | Deterministic checks (regex, field matching) |
Latency needs a rule of its own: measure it where the caller hears it. Coval’s Latency metric does exactly that. It averages the silences of half a second or more on the recording between the caller finishing a turn and the agent starting to answer.
Span timing alone can make an agent look faster than it sounds. In our text-to-speech benchmark (30-day averages to October 28, 2026), Cartesia’s Sonic 3.5 returned its first audio chunk after 160 ms, but the stream opened with 131 ms of silence, so its time to first audio was 291 ms. A first-byte timer showed just over half the wait. Time the first byte and you reward sending anything early, audible or not. The metric improves and the product does not.
Then read the tail. A call has many turns, so even a rare slow reply ends up in a lot of calls. That’s why our August benchmarks AMA put P95 behavior beside the median. Chart Latency at P95 across your calls so the slowest calls don’t vanish into the average. Our guide on how to measure voice AI latency covers budgets and per-stage timing.
How observability catches problems: scoring, judge checks, and alerts
Good metrics only help if they run on the calls that go wrong, and you can’t know in advance which calls those are. So observability scores every call, checks the scorer against human reviewers, and raises an alert when a score crosses a threshold.
Take a failure that hits 1 call in 200, and a manual QA team that listens to 100 random calls. The chance that none of the 100 contains it is 0.995 to the 100th power, or about 61%. That’s why Coval scores every production call against the metrics you choose.
Scoring every call puts the weight on the judge. Our docs say to aim high, around 90% agreement with your reviewers, once they’ve labeled a full sample of calls. In one calibration we published, an AI judge checking whether an outbound agent disclosed that it was an AI showed 89% agreement with human reviewers across 314 conversations. All 35 disagreements were calls where a wrong party picked up or the caller hung up early, so the agent never reached the disclosure. The agent wasn’t broken. The metric was. One added paragraph in the judge prompt fixed it.
Human labels are what caught it. Without them, you can’t tell an agent failure from a judge error. Coval’s human review is built for this: reviewers label calls, and every override and annotation retrains the AI judge.
Alerts put a person on the problem. A Coval alert checks each conversation as it’s scored, and when a metric crosses your threshold, it can notify Slack, email, or a webhook, or send the call to a triage queue, where your reviewers sort flagged calls by failure type, theme, or severity.
Slow drift, like a hosted model changing underneath you, shows up across many calls rather than on any one. Coval dashboards break any metric out by the prompt or model version you attach to each call, so you can tell whether a score moved with one of your releases or while your prompt and model version stayed the same. You can set an alert straight from the dashboard, too.
Voice AI observability vs pre-launch testing: how production failures become tests
Catching a failed call only pays off if the next release doesn’t repeat it, and that’s where observability meets testing. Testing means running simulated calls before launch, with a test caller talking to your agent. Observability scores real calls after. They work best as one loop, with every production failure turned into a test the agent has to pass before it ships again.
In Coval, every metric works on both simulated conversations and live-monitored production calls, so the check that cleared a release is the one watching its traffic.
Here’s how that loop runs on an illustrative call. An agent collects email addresses by phone. A caller spells one out, speech-to-text drops the ”@”, the agent asks again, and the caller hangs up. Nothing errored. Coval’s Agent Repeats Itself metric flags the call, an alert routes it to triage, and the reviewer finds the missing ”@” in the stt span. Once the agent is fixed, the team builds an Audio Upload test case from the call, which plays the caller’s audio exactly as recorded, and adds it to a nightly scheduled run.
Self-driving teams replay real road logs against new driver behavior, and we do the same with call logs, re-simulating production calls against the new agent. Krew, a Coval customer, puts it simply: “Whenever there’s an edge case, we add it into our eval set and run it again.”
The four maturity levels of voice AI observability
Teams get to that loop in stages. Each level below adds one question you can answer about your calls.
| Level | What you have | The question you can answer | Next step |
|---|---|---|---|
| 0 | Uptime and latency dashboards | Is the agent up and fast? | Capture audio, transcripts, and traces for every call |
| 1 | Every call captured | What happened on the call a customer complained about? | Score every call automatically |
| 2 | Every call scored by a judge checked against human reviewers | How many calls failed this week, and why? | Turn each failed call into a scheduled test |
| 3 | Every failure replayed as a test | Will the next release repeat last month’s failures? | Recheck the judge as the agent and its traffic change |
At levels 0 and 1, you hear about failures from customers. Level 2 finds them first. Level 3 keeps them from coming back.
The squatted transfer line passed every check a level 0 setup runs, because nothing at that level asks whether a call worked. Voice AI observability asks that question of every production call. It keeps the audio, transcript, turn timing, tool calls, and outcome, scores the call on metrics you’ve checked against human reviewers, and turns each failure into a test. To see where your agent stands, start a 7-day free trial and upload last week’s calls.
Frequently asked questions
Do I need an observability platform, or can I build it myself?
You can build the capture half yourself: recordings, transcripts, and OpenTelemetry spans are standard plumbing. The evaluating half is where the work piles up: a judge for every metric, calibration against human reviewers, alerting, review queues, and replaying failed calls as tests, all kept in sync with your pre-launch tests. Our build or buy guide walks through that trade-off.
How much does voice AI observability cost?
On Coval’s published plans, Starter is $100 a month for 3,000 upload minutes, with traces kept for 30 days. Growth is $500 a month for 30,000 upload minutes and 90 days of traces. Upload-minute overage is $0.10 a minute on Starter and $0.05 on Growth. Both plans start with a 7-day free trial, and Enterprise starts at $4,500 a month.