LLM Observability with OpenTelemetry: GenAI Semantic Conventions

Traditional observability answers three questions: did the request succeed, how long did it take, and what failed? For LLMs, those questions are necessary but not sufficient. A GPT-4o call can return HTTP 200, finish in 2.3 seconds, and still be completely wrong — hallucinated, off-topic, or financially ruinous.

LLM-specific questions you also need to answer:

• What prompt was sent? (Was it the version you think it was?)
• How many tokens were consumed — prompt vs. completion?
• What did the model return, and did it call a tool?
• What did this cost, per request, per user, per day?
• Which model version ran — did something silently change?

Without a shared vocabulary for these questions, every vendor invents its own attribute names (llm.prompt_tokens vs llm.usage.prompt_tokens vs usage.inputTokens). That's where OTel's GenAI semantic conventions come in.

The OTel GenAI semantic conventions live in the opentelemetry-semantic-conventions spec under gen_ai.*. As of early 2025 they're in Experimental stability but are already widely adopted.

Core attribute groups:

gen_ai.system uses a controlled vocabulary — openai, anthropic, aws_bedrock, google_vertexai, ollama, etc. — so dashboards can group across providers without string-munging.

Each OTel GenAI span gets gen_ai.operation.name to distinguish what kind of work happened:

• chat — a prompt/completion round trip. This is the main span for most LLM calls.
• text_completion — legacy completions API (GPT-3 style).
• embeddings — a call to text-embedding-3-small etc. Records gen_ai.usage.input_tokens but no output tokens.
• create_agent / invoke_agent — for managed agent APIs (Bedrock Agents, Vertex AI Agent Builder).

For tool calls, the spec adds gen_ai.tool.name (the function name) and gen_ai.tool.call.id (the ID the model assigned). The tool execution should be a child span of the chat span that triggered it, which gives you a clean parent-child trace: chat → tool → chat (follow-up turn).

Heads up: prompt and completion content (the actual text) is not in attributes by default — it's captured as span events named gen_ai.content.prompt and gen_ai.content.completion. This is intentional: content can be large and sensitive, so you opt-in per exporter.

The fastest path to OTel traces is the community opentelemetry-instrumentation-openai package (maintained by Traceloop, used by Arize Phoenix and others):

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

# Wire up the provider once at startup
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
)
trace.set_tracer_provider(provider)

# Auto-instrument — patches openai.OpenAI and openai.AsyncOpenAI
OpenAIInstrumentor().instrument(
    enrich_token_usage=True,   # adds gen_ai.usage.* from response headers
    capture_content=True,      # adds gen_ai.content.* events (opt-in!)
)

After this, every client.chat.completions.create() call automatically emits a span with the correct gen_ai.* attributes. No manual span creation needed.

Auto-instrumentation gives you the model telemetry; you need to add business context yourself. The OTel spec has reserved attribute names for this:

from opentelemetry import trace

tracer = trace.get_tracer("myapp")

with tracer.start_as_current_span("handle_user_request") as span:
    span.set_attribute("user.id", user_id)          # who triggered it
    span.set_attribute("gen_ai.request.model", "gpt-4o")  # in case you override
    span.set_attribute("session.id", session_id)    # group multi-turn convos

    response = client.chat.completions.create(...)

    # Cost attribution — compute from token counts + pricing table
    input_cost  = response.usage.prompt_tokens     * 0.0000025
    output_cost = response.usage.completion_tokens * 0.0000100
    span.set_attribute("gen_ai.usage.cost_usd", input_cost + output_cost)
    span.set_attribute("gen_ai.usage.input_cost_usd", input_cost)
    span.set_attribute("gen_ai.usage.output_cost_usd", output_cost)

gen_ai.usage.cost_usd isn't in the official spec yet, but it's the de facto convention used by Arize, Langfuse, and Grafana dashboards. Attach it early and your cost dashboards are free.

At 1,000 LLM calls per minute, recording every span is expensive — both in storage and in latency from the exporter. Three sampling approaches:

Head-based sampling (ParentBased + TraceIdRatio): Decide at trace start whether to record. Simple, but you'll miss rare errors if your sample rate is low.

from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

sampler = ParentBased(root=TraceIdRatioBased(0.10))  # record 10 % of traces
provider = TracerProvider(sampler=sampler)

Tail-based sampling: Buffer spans, then decide after the trace completes. Keeps 100% of error traces. Requires a collector like the OTel Collector with the tailsampling processor. Higher infra cost, much better signal.

Always-on for high-value paths: Sample 10% overall, but force-sample traces where gen_ai.usage.cost_usd > 0.05 or http.route == "/api/premium" using a custom sampler that inspects baggage.

For most teams: start with 10-20% head-based, add tail sampling for errors once volume justifies it.

The OTel Collector is the linchpin. Your app sends OTLP to the Collector; the Collector fans out to whichever backends you use:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }

processors:
  batch: {}
  filter/llm_only:
    spans:
      include:
        match_type: regexp
        attributes:
          - { key: gen_ai.system, value: ".+" }  # only spans with gen_ai.system

exporters:
  otlphttp/phoenix:   { endpoint: "http://phoenix:6006/v1/traces" }
  datadog:            { api:
                          key: ${env:DD_API_KEY}
                          site: datadoghq.com }
  prometheusremotewrite:
    endpoint: "http://mimir:9009/api/v1/push"

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [batch]
      exporters:  [otlphttp/phoenix, datadog]

One instrumented app, three observability backends — no code changes required when you add or swap a backend.

The four major LLM observability platforms each consume OTel differently:

Arize Phoenix — built from the ground up on OTel. Ingests OTLP natively, renders gen_ai.content.prompt events as a prompt/completion view, and aggregates gen_ai.usage.input_tokens + gen_ai.usage.output_tokens into a token dashboard out of the box.

Datadog LLM Observability — has its own ddtrace SDK with LLMObs wrapper, but also ingests OTLP and maps gen_ai.* to its internal model. Adds flamegraph views per model and alert presets for token spike detection.

Grafana (Tempo + Mimir) — generic tracing, but the community Grafana GenAI dashboard (dashboard ID 21693) reads gen_ai.* attributes directly. Pair with a prometheusremotewrite exporter to get per-model token metrics in Mimir.

LangSmith — primarily its own SDK, but as of late 2024 is piloting an OTLP ingestion endpoint. If your traces carry gen_ai.operation.name = chat and session.id, LangSmith reconstructs multi-turn runs automatically.

The standard wins here: if you write gen_ai.*, you can switch backends without re-instrumenting.

Cost attribution at scale requires more than summing token counts. Two patterns that work:

Pattern 1 — Attribute-based grouping in Grafana/Prometheus. Emit gen_ai.usage.cost_usd as a span attribute, then use the Collector's spanmetrics processor to turn it into a Prometheus histogram with labels gen_ai.system, gen_ai.request.model, and user.id:

processors:
  spanmetrics:
    metrics_exporter: prometheusremotewrite
    dimensions:
      - name: gen_ai.system
      - name: gen_ai.request.model
      - name: user.id
    histogram:
      explicit:
        buckets: [0.001, 0.01, 0.05, 0.10, 0.50, 1.00]

Now you can query sum(gen_ai_usage_cost_usd_sum) by (user_id) for a per-user cost report.

Pattern 2 — Baggage propagation. Set tenant_id and feature_flag in OTel Baggage at the edge, and they'll flow through every downstream span automatically — including your LLM spans. No need to pass them manually into every LLM call.

Lessons from teams running this in production:

Gotcha 1 — Capturing content in prod. capture_content=True in the instrumentor logs raw prompts and completions. That's PII-in-logs unless you have a scrubbing processor. Use it in dev/staging; in prod, capture only tokens + finish reason, and add content selectively via evaluated samples.

Gotcha 2 — Double-counting tokens in multi-turn. Each turn of a conversation sends the full history as context. Your per-turn input_tokens includes all previous messages. Don't sum across turns to get "total input tokens" — it overcounts. Track cumulative cost separately via a session.total_cost_usd baggage value.

Gotcha 3 — Model version drift. gen_ai.request.model is what you asked for; gen_ai.response.model is what ran. OpenAI auto-upgrades gpt-4 to the latest snapshot. Compare them in your dashboard — divergence means your prompts are running on a model you haven't evaluated.

Gotcha 4 — Missing parent span. If your LLM call isn't inside a parent span, each call is its own root trace. You lose the ability to see which user request caused the LLM call. Always wrap LLM calls inside a named span for the outer operation.

Before OTel GenAI conventions, every vendor had its own attribute names, SDK, and export format. Switching from LangSmith to Phoenix meant re-instrumenting. Comparing token costs across two providers required custom ETL.

With gen_ai.*:

• Vendor portability — instrument once, export anywhere via the Collector.
• Shared tooling — community dashboards, alerting rules, and samplers work across stacks.
• Multi-model comparisons — gen_ai.request.model and gen_ai.system let you A/B test GPT-4o vs. Claude 3.5 Sonnet in the same dashboard with zero extra code.
• Compliance evidence — a standardized audit trail of what prompts were sent, by whom, and when, without building custom logging.

The spec is still evolving (Experimental stability as of 2025), so expect attribute names to stabilize over the next 12 months. Pin your semantic-conventions package version and watch the changelog — but the investment in OTel instrumentation today pays dividends regardless of which backend wins the market.