Skip to main content
An online eval scores real production traffic after it happens, instead of a prepared test set. In Agnost AI, online evals are built from parts you already use: rules are the evaluators, violations are the failing verdicts, alerts are the thresholds, and Auto Improve is the follow-up action. The Agnost app labels this workflow Violations.

​
How Agnost AI maps to online eval concepts

​
When judging runs

Agnost AI waits until a conversation has been idle for 15 minutes, so a step that happens later in the conversation is not reported as missing. The judge then reads the conversation from start to finish against every active rule in one pass. Conversations are processed in batches, so results are not real time. Use online evals to review behavior, and keep checks that must block a response in your application code. When you create a rule, turn it back on, or change its description, Agnost AI also judges your most recent 1,000 conversations against it. You can read real flagged examples before you set an alert threshold.

​
Set up your first online eval

1

Send complete conversations

Instrument your agent with an SDK or OpenTelemetry. Include every user and assistant message. The judge reads the conversation’s top-level turns; tool calls and sub-agent spans nested under a turn are left out. If a rule depends on an outcome, such as a refund settling, record that outcome as its own event.
2

Write one narrow rule

Open Violations and add a rule for one behavior, for example “State the cancellation policy before completing a cancellation.” Avoid broad rules such as “be helpful.” Rules that depend on information Agnost AI never receives cannot be judged.
3

Read the evidence

Open Evidence on the rule and read flagged conversations. Also read some conversations that were not flagged. If you disagree with the verdicts, make the rule more specific and state its exceptions.
4

Add an alert

Choose Alert on the rule. Set a count threshold, lookback window, schedule, and Slack or email destination. See Alerts.
5

Close the loop

With Auto Improve connected, a fired alert starts a run that opens a pull request for review. About an hour after the pull request is merged, Agnost AI re-checks the alert and records whether the condition improved.

​
Query results from code and AI clients

Rule statistics, violations, and timelines are available through the Analysis API with an API key, and through the hosted MCP server from compatible AI clients.

​
What to know

  • Verdicts are pass/fail flags, not graded scores.
  • Nested tool calls are tracked with success rates, latency, and error distributions rather than judged by the LLM. See Tool calls.
  • A violation is a classified finding, not enforcement. Keep critical safety and authorization checks in application code.
  • Agnost AI does not run offline datasets or CI eval gates. Pair it with your existing test suite, and turn confirmed production failures into regression tests there.

​
Next steps