> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agnost.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Online evals

> Run online evaluations on production agent conversations with plain-English rules, an LLM judge, alerts, and Auto Improve

An **online eval** scores real production traffic after it happens, instead of a
prepared test set. In Agnost AI, online evals are built from parts you already
use: **rules** are the evaluators, **violations** are the failing verdicts,
**alerts** are the thresholds, and **Auto Improve** is the follow-up action.

The Agnost app labels this workflow **Violations**.

## How Agnost AI maps to online eval concepts

| Online eval concept | In Agnost AI |
| - | - |
| Evaluator or rubric | A rule (SOP), written in plain English on the **Violations** page |
| Unit judged | The whole conversation; verdicts point to the assistant messages that broke a rule |
| Trigger | The conversation has been idle for 15 minutes |
| Judge | An LLM judge run by Agnost AI; no provider key needed |
| Population | Every analyzed conversation, up to your plan's monthly event limit; there is no sampling setting |
| Verdict | A violation flag on the message, with the conversation as evidence |
| Rate | Violations compared with analyzed conversations, per rule and over time |
| Slice | Intents, which label what each user was trying to do |
| Threshold and notification | An [alert](https://docs.agnost.ai/using-alerts.md) on a rule's violation count, sent to Slack or email |
| Action after failure | [Auto Improve](https://docs.agnost.ai/using-improvements.md) opens a pull request and re-checks the alert after merge |

## When judging runs

Agnost AI waits until a conversation has been idle for 15 minutes, so a step that
happens later in the conversation is not reported as missing. The judge then reads
the conversation from start to finish against every active rule in one pass.
Conversations are processed in batches, so results are not real time. Use online
evals to review behavior, and keep checks that must block a response in your
application code.

When you create a rule, turn it back on, or change its description, Agnost AI
also judges your most recent 1,000 conversations against it. You can read real
flagged examples before you set an alert threshold.

## Set up your first online eval

**1. Send complete conversations**

Instrument your agent with an [SDK](https://docs.agnost.ai/sdks.md) or [OpenTelemetry](https://docs.agnost.ai/otel.md). Include
every user and assistant message. The judge reads the conversation's top-level
turns; tool calls and sub-agent spans nested under a turn are left out. If a
rule depends on an outcome, such as a refund settling, record that outcome as
its own event.

**2. Write one narrow rule**

Open **Violations** and add a rule for one behavior, for example "State the
cancellation policy before completing a cancellation." Avoid broad rules
such as "be helpful." Rules that depend on information Agnost AI never receives
cannot be judged.

**3. Read the evidence**

Open **Evidence** on the rule and read flagged conversations. Also read some
conversations that were not flagged. If you disagree with the verdicts, make
the rule more specific and state its exceptions.

**4. Add an alert**

Choose **Alert** on the rule. Set a count threshold, lookback window, schedule,
and Slack or email destination. See [Alerts](https://docs.agnost.ai/using-alerts.md).

**5. Close the loop**

With [Auto Improve](https://docs.agnost.ai/using-improvements.md) connected, a fired alert starts a run
that opens a pull request for review. About an hour after the pull request is
merged, Agnost AI re-checks the alert and records whether the condition
improved.

## Query results from code and AI clients

Rule statistics, violations, and timelines are available through the
[Analysis API](https://docs.agnost.ai/high-level-apis.md) with an API key, and through the
[hosted MCP server](https://docs.agnost.ai/agnost-mcp-server.md) from compatible AI clients.

## What to know

* Verdicts are pass/fail flags, not graded scores.
* Nested tool calls are tracked with success rates, latency, and error
  distributions rather than judged by the LLM. See [Tool calls](https://docs.agnost.ai/using-tool-calls.md).
* A violation is a classified finding, not enforcement. Keep critical safety and
  authorization checks in application code.
* Agnost AI does not run offline datasets or CI eval gates. Pair it with your
  existing test suite, and turn confirmed production failures into regression
  tests there.

## Next steps

* [Violations](https://docs.agnost.ai/using-violations.md): configure rules and review evidence.
* [Alerts](https://docs.agnost.ai/using-alerts.md): monitor important rules and thresholds.
* [Improvements](https://docs.agnost.ai/using-improvements.md): turn repeated violations into pull requests.
* [What are online evals?](https://agnost.ai/blog/what-are-online-evals/): the concept, compared with offline evals.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.