Choose a setup method
Start with one method for a call path. Combining SDK tracking with framework
OpenTelemetry on the same call can create duplicate interactions.
Recommended: Agnost AI skill
Recommended. Choose this path when you want your coding agent to inspect the
project, select a supported transport, make the smallest instrumentation change,
send a test event, and verify that it reached Agnost AI.
Manual setup: Agnost AI SDK
agnost.shutdown() once when the application exits, not after each request.
Manual setup: OpenTelemetry
The OpenInference LlamaIndex instrumentation emits an OpenTelemetry span for every agent run, workflow step, model call, and tool call. Send those spans to Agnost AI over OTLP, and wrap each turn in one short span that carries the user’s message and the agent’s answer.LlamaIndex also ships
llama-index-observability-otel. Its spans carry raw workflow events with no model, message, or token attributes, so Agnost AI does not ingest them. Use OpenInference.1. Install
llama-index 0.14.25, openinference-instrumentation-llama-index 4.7.6, and opentelemetry-sdk 1.45.2 on Python 3.12.
2. Point OpenTelemetry at Agnost AI and run an agent
LlamaIndexInstrumentor().instrument()traces agents, workflows, query engines, retrievers, LLMs, and tools.using_attributes(user_id=..., session_id=...)writesuser.idandsession.idonto every span LlamaIndex emits inside the block. Every turn with the samesession_idlands in the same Agnost AI conversation.- The
agent_turnspan becomes the root of each trace. Agnost AI builds the conversation transcript from each trace’s root span, and the root span OpenInference writes foragent.run()holds the serialized workflow state, not the user’s message. The wrapper puts the message and the final answer there instead. It only adds span attributes, so it does not change how the agent runs.
TracerProvider and call instrument() once at startup, call run_turn per request, and call provider.shutdown() on graceful shutdown.
What appears in Agnost AI
- Conversations: one per
session_id. Agnost AI keeps the raw session ID asoriginal_conversation_idin the conversation metadata. - Users: the
user_idyou pass tousing_attributes. - Messages: the user’s message and the agent’s answer for each turn, from the
agent_turnspan. - Events: the agent run, each workflow step (
chain), each model call (llm), and each tool call (tool) in a nested tree. Model calls include the model name, input and output messages, tool calls, and token counts. Tool calls include the tool name, arguments, and result.
Verify
Run the script, wait one to five minutes for ingestion, then open Conversations in Agnost AI. Confirm the conversation shows youruser_id, a user message and an assistant answer, and a get_order_status tool event with arguments and a result.
Troubleshooting
- Nothing arrives: confirm the exporter’s headers contain
X-Agnost-Org-ID, and that the script callsprovider.shutdown()orprovider.force_flush()before it exits.BatchSpanProcessorexports in the background. - The user message is a large JSON blob of workflow state: the turn ran outside the
agent_turnspan. Callagent.run()insiderun_turn. - User or conversation is missing: await
agent.run()inside theusing_attributesblock. - Model calls have no token counts: OpenAI returns usage on streamed responses only when asked. Keep
stream_options={"include_usage": True}inadditional_kwargs, or setstreaming=Falseon the agent.
References
Next steps
- Conversations: review the complete LlamaIndex session.
- Events: inspect agent, model, and tool activity.
- Intents: organize production conversations by what users wanted.