AI agent observability: how to know if yours is actually working

An AI agent can answer fast, with polished text, and still be wrong half the time. Without observability, nobody notices until a customer complains or a sales lead disappears. AI agent observability is the set of metrics and logs that show, in real time, whether an agent is actually doing its job right.

What AI agent observability actually means

Observability is different from “the agent is up.” It means knowing, with data, three things: what the agent answered, what that answer cost, and whether it solved the user’s problem.

In traditional software, observability means logs, metrics, and traces. For AI agents, the logic is similar, but the signals change: response latency, cost per token, escalation rate to a human, and above all, whether the answer was correct.

The catch is that an agent never “breaks” the way a server does. It keeps answering. Just wrong, incomplete, or off-tone. Without observability, that kind of failure is invisible.

Why “it’s running” doesn’t tell you anything

Teams often measure an AI agent by uptime: it responded, so everything must be fine. That hides the real problems.

An agent can have 100% uptime and still:

• Hallucinate a refund policy that doesn’t exist.
• Take 40 seconds to answer a simple question, driving the customer to give up.
• Cost 5x more per conversation than it should, because it’s using the wrong model for the task.
• Escalate to a human in 60% of conversations, which means the automation isn’t automating anything.

None of those problems show up on an “is it online” dashboard. They all show up when you look at the right metrics.

A real example of “everything looked fine”

A WhatsApp support agent replied in under 3 seconds and had 99.8% uptime for the month. On paper, perfect.

The problem only surfaced when someone read a sample of 30 conversations: in 6 of them, the agent had invented a delivery deadline that didn’t exist in the system. Nobody had noticed, because no uptime or latency metric captures “the answer was wrong.” Uptime stayed at 99.8%. The hallucination rate, if anyone had measured it, was sitting at 20%.

That’s the most common pattern: the infrastructure dashboard says everything is fine, and the quality dashboard doesn’t exist at all.

The metrics that actually matter

Four numbers tell most of the story of an agent in production.

1. Latency per response

Time between the user’s question and the agent’s answer. Past a few seconds, the conversation starts to feel stuck, especially on channels like WhatsApp, where the expectation is a near-instant reply.

2. Cost per conversation

How much each full interaction costs in tokens. Different models have very different prices, and the same task can cost 10x more depending on which model the agent uses. Without measuring this per conversation, cost only shows up on the monthly bill, too late to fix.

3. Escalation rate to a human

How many conversations end with “let me transfer you to an agent.” A high rate isn’t a sign of care, it’s a sign the agent isn’t resolving what it’s supposed to resolve on its own.

4. Error and hallucination rate

The hardest metric to capture and the most important one. How many answers were factually wrong, invented information, or drifted from what the knowledge base actually says.

How to build a dashboard without turning it into an engineering project

The temptation is to build an observability system from scratch: custom logging, a Grafana dashboard, Slack alerts. That works, but it takes weeks and becomes one more system to maintain.

The faster path for most companies is a platform that already logs this natively:

1. Log every conversation, with the model used, response time, and token cost.
2. Flag every escalation, whenever the agent hands the conversation to a human.
3. Sample review, where someone on the team reads a slice of conversations every week to catch what the metrics alone miss.
4. Automatic alerts when a number drifts from baseline, like average cost climbing 30% from one week to the next.

It doesn’t need to be sophisticated in month one. It needs to exist. Most agents that “quietly went wrong” had zero of these four in place.

Building this from scratch usually takes two to six weeks of engineering work: instrumenting every model call, building the dashboard, wiring up alerts. That’s real work, but it’s work that already ships out of the box on platforms that treat observability as part of the product, not as a separate internal project.

Warning signs your agent is quietly getting worse

A few patterns show up before they turn into a customer complaint:

Cost per conversation creeping up. Usually a sign the agent is burning more tokens to reach the same answer, or switched models without anyone noticing.

Escalation climbing month over month. The knowledge base went stale, or a new type of question showed up that the agent can’t answer.

Response time swinging wildly. If latency is unstable (fast sometimes, slow other times), it’s usually a sign the model or integration is under load or misconfigured.

Answers getting vaguer. When the agent starts giving generic replies instead of specific ones, it’s a sign it isn’t finding the right information in the knowledge base.

None of these signals is dramatic on its own. Together, they point to an agent that needs attention before the customer notices first.

Observability is not the same thing as audit trails

The two are easy to mix up. Audit trails answer “who said what to whom,” built for compliance and governance: tracing every conversation, proving sensitive data didn’t leak, showing conformance when someone asks.

Observability answers “is the agent doing a good job,” built for performance: speed, cost, accuracy.

You need both. But they’re different teams asking different questions. A company can have perfect audit trails (every log stored) and zero observability (nobody looks at the log to see if the agent is any good).

In practice, whoever owns compliance tends to ask for audit trails. Whoever owns the product or the support team tends to need observability. If only one of those groups is asking for it at your company, the other one is probably missing.

A practical checklist to get started

If you’re measuring nothing today, start here, in this order:

1. Pick the four metrics above and make sure your platform logs all four per conversation.
2. Set an acceptable target for each one (e.g., latency under 5 seconds, escalation under 20%).
3. Block one fixed day a week to check the dashboard and read a sample of real conversations.
4. Set up a simple alert for when a number drifts outside the acceptable range.

That already fixes most of the “we found out too late” problems.

How often to check the dashboard

There’s no single right answer, but this cadence works as a baseline:

• Daily, but only for automatic alerts (cost drifting out of range, latency spiking). Nobody needs to open the dashboard every day if an alert already flags what’s abnormal.
• Weekly, for a sample review of conversations and a read of the four core metrics. That’s the minimum rhythm to catch a trend before it becomes a problem.
• Monthly, for total cost trends and model comparisons, feeding into budget decisions and model switches.

Companies that only check the dashboard once a month find out about problems a month late. Companies that only check when a customer complains don’t have observability, they have damage control.

The fastest way to get this in place

Building observability from scratch is possible, but it burns a lot of engineering time. SquadOS already logs latency, cost per conversation, and escalation natively, with dozens of models available and multimodal processing that cuts token cost by up to 95%. You can switch models, compare real cost, and see what the agent is actually doing, without building your own infrastructure.