Skip to main content

Choose a setup method

Start with one method for a call path. Combining SDK tracking with framework OpenTelemetry on the same call can create duplicate interactions.
Recommended. Choose this path when you want your coding agent to inspect the project, select a supported transport, make the smallest instrumentation change, send a test event, and verify that it reached Agnost AI.
Install the skill once in your coding-agent environment:
Run the framework-specific prompt below from the application root. Review the resulting diff before deploying it.

Manual setup: Agnost AI SDK

Call agnost.shutdown() once when the application exits, not after each request.

Manual setup: OpenTelemetry

The OpenInference LlamaIndex instrumentation emits an OpenTelemetry span for every agent run, workflow step, model call, and tool call. Send those spans to Agnost AI over OTLP, and wrap each turn in one short span that carries the user’s message and the agent’s answer.
LlamaIndex also ships llama-index-observability-otel. Its spans carry raw workflow events with no model, message, or token attributes, so Agnost AI does not ingest them. Use OpenInference.

1. Install

Tested with llama-index 0.14.25, openinference-instrumentation-llama-index 4.7.6, and opentelemetry-sdk 1.45.2 on Python 3.12.

2. Point OpenTelemetry at Agnost AI and run an agent

What each piece does:
  • LlamaIndexInstrumentor().instrument() traces agents, workflows, query engines, retrievers, LLMs, and tools.
  • using_attributes(user_id=..., session_id=...) writes user.id and session.id onto every span LlamaIndex emits inside the block. Every turn with the same session_id lands in the same Agnost AI conversation.
  • The agent_turn span becomes the root of each trace. Agnost AI builds the conversation transcript from each trace’s root span, and the root span OpenInference writes for agent.run() holds the serialized workflow state, not the user’s message. The wrapper puts the message and the final answer there instead. It only adds span attributes, so it does not change how the agent runs.
In a long-running server, create the TracerProvider and call instrument() once at startup, call run_turn per request, and call provider.shutdown() on graceful shutdown.

What appears in Agnost AI

  • Conversations: one per session_id. Agnost AI keeps the raw session ID as original_conversation_id in the conversation metadata.
  • Users: the user_id you pass to using_attributes.
  • Messages: the user’s message and the agent’s answer for each turn, from the agent_turn span.
  • Events: the agent run, each workflow step (chain), each model call (llm), and each tool call (tool) in a nested tree. Model calls include the model name, input and output messages, tool calls, and token counts. Tool calls include the tool name, arguments, and result.

Verify

Run the script, wait one to five minutes for ingestion, then open Conversations in Agnost AI. Confirm the conversation shows your user_id, a user message and an assistant answer, and a get_order_status tool event with arguments and a result.

Troubleshooting

  • Nothing arrives: confirm the exporter’s headers contain X-Agnost-Org-ID, and that the script calls provider.shutdown() or provider.force_flush() before it exits. BatchSpanProcessor exports in the background.
  • The user message is a large JSON blob of workflow state: the turn ran outside the agent_turn span. Call agent.run() inside run_turn.
  • User or conversation is missing: await agent.run() inside the using_attributes block.
  • Model calls have no token counts: OpenAI returns usage on streamed responses only when asked. Keep stream_options={"include_usage": True} in additional_kwargs, or set streaming=False on the agent.

References

Next steps

  • Conversations: review the complete LlamaIndex session.
  • Events: inspect agent, model, and tool activity.
  • Intents: organize production conversations by what users wanted.