mcpkit

Observability

SEP-414 OpenTelemetry tracing across the server, client, tasks, events, and the Apps bridge.

Source: docs/SEP_414_OTEL.md

SEP-414: OpenTelemetry Trace Context Propagation

Status: Phases 1, 2, 3, and 4 landed. Polished walkthroughs (Phase 5)
and the conformance suite (issue 429) still to come.

This document is the wiring guide for mcpkit’s OpenTelemetry surface. It
covers what the shipped phases give you, what to expect once you wire a
real TracerProvider, and where the remaining work lives on
issue 312.

What Phase 1 ships

Phase 1 lands the dependency-free contract surface in core/:

  • core.TracerProvider, core.Span, core.Attribute — the minimal
    tracing seam mcpkit components consume. The default,
    core.NoopTracerProvider, performs no allocation and emits no spans.
  • core.TraceContext — the W3C traceparent / tracestate pair as
    propagated on the MCP wire.
  • core.ExtractTraceContext / core.InjectTraceContext — read/write
    W3C Trace Context from/to an MCP _meta map, with strict
    W3C-version-00 validation on extract.
  • core.WithTraceContext / core.TraceContextFromContext —
    context.Context plumbing.
  • core.BaseContext.TraceContext() — accessor exposed on every typed
    handler context (ToolContext, PromptContext, ResourceContext,
    MethodContext) so handlers can read the active trace context
    without import gymnastics.

The MetaKeyTraceparent / MetaKeyTracestate constants pin the wire
keys (bare traceparent and tracestate, matching W3C HTTP header
names, not under the io.modelcontextprotocol/ namespace, which is
reserved for MCP-defined fields).

Phase 1 introduces no behavior change: nothing in mcpkit calls
TracerProvider.StartSpan yet, and no transport reads or writes the
_meta trace keys. The surface exists so downstream work can be
written and reviewed against a stable contract.

What Phase 2 ships

Phase 2 wires the server-side propagation surface:

  • server.WithTracerProvider(tp core.TracerProvider) Option — install a
    tracer. Default and core.NoopTracerProvider{} both skip the trace
    middleware entirely so the zero-overhead path stays untouched.
  • An internal traceMiddleware wraps every JSON-RPC dispatch in an
    inbound span, positioned OUTERMOST so user-registered middleware
    contributes to the recorded latency. Span attributes:
    • mcp.method on every span
    • mcp.tool.name on tools/call
    • mcp.session.id when a session is attached
    • mcp.error.code + Span.RecordError on JSON-RPC errors
    • mcp.tool.is_error="true" on tools/call results carrying IsError
  • Inbound trace context resolution: params._meta.traceparent wins
    over the out-of-band TraceContext bridged through context.Context.
    Malformed traceparent values are dropped per the W3C “MUST NOT
    forward” rule. The span still emits with no parent.
  • The streamable HTTP transport bridges the W3C traceparent /
    tracestate HTTP headers (SEP-2028) into context.Context once at
    the POST entry point. All downstream dispatch paths (sync JSON, SSE,
    batch, stateless, initialize) inherit the bridge.
  • Outbound _meta.traceparent / _meta.tracestate are injected on
    every server-to-client message, covering notifications (logging, progress,
    resource updates, custom) and server-to-client requests (legacy push
    for sampling/elicitation/roots). The wrap sits OUTSIDE any
    user-registered NotifyInterceptor / RequestInterceptor, so those
    interceptors observe the same wire form the client receives.

Wiring is a single line in your server bootstrap:

srv := server.NewServer(info,
    // ...your other options...
    server.WithTracerProvider(myTracerProvider),
)

myTracerProvider must implement core.TracerProvider. Until the
Phase 4 ext/otel/ adapter ships, the only options that emit spans are
custom implementations. The in-tree core.NoopTracerProvider
intentionally treats “spans” as no-ops.

What Phase 4 ships

Phase 4 lands the OpenTelemetry SDK adapter as a new sub-module under
ext/otel/ (separate go.mod, mirroring ext/auth/ / ext/tasks/ /
ext/ui/). Wire it in with one line:

import (
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"

    "github.com/panyam/mcpkit/server"
    mcpotel "github.com/panyam/mcpkit/ext/otel"
)

exp, _ := stdouttrace.New()
otelTP := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp))
defer otelTP.Shutdown(ctx)

srv := server.NewServer(info,
    server.WithTracerProvider(mcpotel.NewProvider(otelTP)),
)

Behavior:

  • The adapter parses core.TraceContext from ctx into an OTel
    SpanContext so the inbound parent flows into the SDK’s standard
    parent-tracking machinery.
  • After StartSpan, it formats the new span’s SpanContext as a W3C
    traceparent and re-attaches via core.WithTraceContext, so the
    Phase 2 outbound _meta injection wraps stamp every server-to-client
    message with the child span’s traceparent (not the inbound
    parent’s).
  • core.Attribute{Key, Value string} maps to attribute.String(...);
    RecordError(err) emits an OTel “exception” event AND sets the span
    status to codes.Error; End() is idempotent.
  • WithInstrumentationName(...) overrides the default
    instrumentation library name ("github.com/panyam/mcpkit/server").

Verification:

  • just test-otel — adapter unit tests run against the real OTel SDK
    in-memory exporter; reads back sdktrace.ReadOnlySpan shapes.
  • just test-otel-example — smoke test for examples/otel/stdout/
    asserting the exporter actually prints the expected span set.
  • go run examples/otel/stdout/... — runnable demo, prints spans as
    JSON on stdout. No collector required.

See ext/otel/README.md for the API reference
and examples/otel/stdout/README.md
for the walkthrough.

What Phase 3 ships

Phase 3 lands the symmetric client-side surface:

  • client.WithTracerProvider(tp core.TracerProvider) — install a
    tracer. Default and core.NoopTracerProvider{} both skip the install
    (zero overhead on the unconfigured path).
  • Outbound Client.Call is wrapped in a span via a ClientMiddleware
    installed as the OUTERMOST entry. User middleware (auth retry,
    header injection) runs inside the span so its latency is captured.
    Span attributes:
    • mcp.method on every span
    • mcp.tool.name on tools/call
    • mcp.client.session.id when a session exists (note client.
      namespace, which lets operators filter by client- vs
      server-emitted)
    • mcp.error.code + Span.RecordError on *RPCError
  • Outbound params gain _meta.traceparent / _meta.tracestate
    derived from the active span via core.InjectTraceContextIntoParams
    (promoted from the server-internal helper so both wires apply the
    same precedence rule, where explicit caller-set values win).
  • Inbound server-to-client requests
    (sampling/createMessage, elicitation/create, roots/list):
    params._meta.traceparent is extracted via
    core.ExtractTraceContextFromParams, attached to ctx via
    core.WithTraceContext, and a wrap span named after the request
    method is emitted with the inbound trace context as parent. The
    handler observes the inbound TC via core.TraceContextFromContext(ctx)
    and can use it as a parent for its own spans.

Wiring is a single line in your client bootstrap:

c := client.NewClient(url, info,
    // ...your other options...
    client.WithTracerProvider(myTracerProvider),
)

myTracerProvider is the same core.TracerProvider you’d hand the
server side via server.WithTracerProvider. The Phase 4 ext/otel/
adapter (PR 652) is the in-tree implementation.

What comes next

Tracked phases on issue 312:

  • P5 — polished examples. examples/otel/jaeger/ +
    examples/otel/otlp/ end-to-end walkthroughs with collector and UI
    screenshots. The minimal examples/otel/stdout/ already covers
    smoke-verification.
  • Conformance suite testconf-otel — issue 429.
  • P6 — tracing across surfaces. Umbrella issue 663. See below.

Adjacent: W3C Baggage propagation + HTTP forward helper (issue 739)

W3C Baggage is a separate W3C standard from W3C Trace Context
(traceparent / tracestate) that propagates arbitrary key=value
pairs alongside trace identity. Operators typically use it for
tenant_id, user_id, feature flags, A/B-test buckets. mcpkit
propagates it symmetrically to the trace context surface so handlers
can ctx.Baggage() and stamp it onto outbound work without
re-implementing the W3C parsing rules.

core.HTTPForwardTransport(base http.RoundTripper) wraps an
http.Client’s Transport so a tool handler’s downstream HTTP
calls automatically carry the active traceparent / tracestate /
baggage headers. This closes the end-to-end observability loop for
handlers that call third-party APIs. Without it, the trace stops at
the MCP boundary and resumes on the downstream side as a fresh
trace.

These features ship independently of upstream SEP-2028 (which
adds a configurable headerGroups API on top of this same
foundation). Tracking the upstream spec on issue 739; the W3C
standards themselves are stable and ship today.

Adjacent: metrics seam (issue 7)

The core.MeterProvider seam mirrors the SEP-414 core.TracerProvider
shape for metrics. Metrics don’t cross the wire, so no SEP, but the
library still needs a dependency-free seam so the base module can emit
measurements without dragging the OTel metrics SDK in. Wire it via
server.WithMeterProvider; the OTel adapter lives at
mcpotel.NewMeterProvider(otelMP). Canonical instruments emitted from
the server dispatch path (mcp.tool.calls, mcp.jsonrpc.errors,
mcp.tool.duration, mcp.sessions.active) share the same attribute
vocabulary as the SEP-414 spans, and exemplars are wired by default so
Grafana’s metric → trace pivot works without per-call configuration.
See ext/otel/README.md § Metrics for the wiring snippet and
server/metrics_middleware.go for the dispatch-side instrumentation.

Agent Runner metrics — landed (issue 1023)

The agent.Runner emits through the same core.MeterProvider seam, the
metrics sibling of its SEP-414 spans. RunnerConfig.MeterProvider opts
it in; nil or core.NoopMeterProvider is zero overhead. Instruments are
built once in NewRunner (metrics.go) and recorded at the same points the
spans are (runner.go): turn end and each tool call. Those files live in
chakra now, not this repo.

Instrument Kind Attributes Recorded
agent.turns counter agent.finish_reason one per completed turn
agent.turn.duration histogram (s) — turn wall-clock
agent.steps counter — model round-trips, at turn end
agent.tokens counter direction (input/output) tokens per turn
agent.tool.calls counter tool, status one per tool call
agent.tool.duration histogram (s) tool tool-call wall-clock

status is the terminal outcome: ok, error, tool_error (the
handler returned an error result), denied (approval policy), cancelled
(per-call cancel), or unavailable (server down this turn). A deferred
defer in callTool records exactly one point per call regardless of
which outcome path returns, so the counter never double-counts or misses.

The host threads it end to end: host.WithMeterProvider sets it on the
main Runner and every sub-agent persona; agentchat builds it from the
same --exporter/--otlp-endpoint decision as the tracer and logger
(chakra’s surfaces/chat/telemetry_setup_meter.go, the meter sibling of
SetupTelemetry/SetupLogs). The OTLP path lands in Mimir; the
mcpkit — agent Grafana dashboard
(docker/observability/grafana/provisioning/dashboards/files/mcpkit-agent.json)
charts turn rate, latency, token throughput, and tool failure ratio out
of the box. Prometheus renames apply the usual OTel conventions: dots to
underscores, a _total suffix on counters, and the unit suffix on
histograms (agent_turn_duration_seconds_bucket).

P6 — tracing across surfaces (auth / tasks / apps)

P1–P5 instrumented the dispatch spine: every JSON-RPC method gets one
inbound span plus W3C wire propagation, for free. P6 extends tracing into
the work each surface does that the single dispatch span doesn’t cover.

Is this a SEP-414 gap? No. Separate three layers:

Layer Owner Status
Wire contract (the _meta keys, format, precedence, SEP-2028 bridge) SEP-414 — normative, for cross-SDK interop complete
Local span richness (count, names, attributes, links) the SDK — latitude; invisible to the peer mcpkit-completeness work (P6)
Adjacent-transport hops (events bus, apps Bridge) mostly the SDK; the Apps Bridge may belong in the Apps spec open

Span richness never crosses the wire, so a SEP can’t (and shouldn’t)
mandate it. The one genuine spec question is the apps Bridge
(postMessage): if cross-SDK app-tracing interop is a goal, whether
traceparent rides the bridge envelope belongs in the ext-ui / Apps
spec, not SEP-414.

Two categories of “work the spine misses”:

  • (a) escapes the span — async / out-of-band on a non-MCP transport:
    tasks background execution (#659), events cross-replica bus
    (#642 / #629).
  • (b) inside the span, not broken out / enriched — auth validation
    sub-spans + principal attributes (#658).

The two categories map cleanly onto the contract:

  • New propagation surfaces need no new contract — core.ExtractTraceContext
    / InjectTraceContext already operate on any map[string]any, so the
    apps Bridge (#660) and the events bus reuse them verbatim.
  • Enrichment surfaces reveal the only two P1 gaps, neither of which the
    spine needed: core.SpanFromContext (active-span accessor, #661;
    unblocks auth attributes) and span links on core.Span (#662;
    unblocks task lifecycle).

Active-span accessor — landed (issue 661)

The first contract gap is closed: core.SpanFromContext(ctx) core.Span
returns the currently-active mcpkit Span (or a no-op Span when none is
attached). Sibling helper core.WithActiveSpan(ctx, span) Context is what
TracerProvider adapters call after StartSpan to publish the span. The
in-tree NoopTracerProvider and the ext/otel adapter both follow the
pattern, so the contract holds regardless of which provider is wired.

The intended use is decorating the dispatch span, not nesting a child:

// inside a middleware or handler, no ext/otel import needed:
span := core.SpanFromContext(ctx)
span.SetAttribute("mcp.auth.principal", claims.Subject)
span.SetAttribute("mcp.auth.method", "jwt")

Typed handler contexts gain a Span() accessor that delegates to
SpanFromContext:

func myTool(ctx core.ToolContext, req core.ToolRequest) (core.ToolResponse, error) {
    ctx.Span().SetAttribute("mcp.tool.cache.hit", "true")
    // ...
}

The accessor never returns nil, so call sites that always-decorate work
correctly whether or not a TracerProvider is configured. Issue 658
(ext/auth attributes) and any future “enrich the active span” use case
consume this surface without crossing the core/-only boundary.

Span links — landed (issue 662)

The second contract gap is closed: span links are now expressible
dep-free, with the OTel-aligned shape that includes per-link
attributes. Two entry points:

// Option at start (call site knows all links upfront):
links := []core.Link{
    {TraceContext: tc1, Attributes: []core.Attribute{{Key: "link.kind", Value: "originated-from"}}},
    core.LinkFromTraceContext(tc2),
}
ctx, span := core.StartSpanLinked(tp, ctx, "task.execute", links, attrs...)

// Mid-flight (links discovered during span execution):
span := core.SpanFromContext(ctx)
span.AddLink(core.Link{TraceContext: tc, Attributes: ...})

core.Link mirrors the OpenTelemetry spec Link definition: a
TraceContext (the upstream identity, same shape carried on the MCP
wire via _meta) plus optional per-link attributes. Per-link
attributes are how observability backends render the link UI.
Jaeger / Tempo / Honeycomb all show link.kind=... semantically
rather than as generic span attributes.

Capability widening pattern: core.LinkedTracerProvider is a
sibling of TracerProvider that adds the StartSpanLinked method;
the core.StartSpanLinked(tp, ...) package-level helper type-asserts
and falls back to plain StartSpan when the provider doesn’t
implement the wider interface, dropping the links but still emitting the span.
This kept the base TracerProvider interface unchanged so non-tracing
test fakes and the default NoopTracerProvider didn’t need any
churn. The ext/otel adapter implements both interfaces and maps
each core.Link to an OTel trace.Link via the existing
traceContextToSpanContext helper.

Invalid links are silently dropped, both at StartSpanLinked
(filtered before passing to the OTel SDK) and at Span.AddLink
(short-circuited inside the wrapper). Defensive call sites can build
link slices from raw inputs (e.g., core.ExtractTraceContext outputs
that might be zero) without pre-filtering. Calling AddLink after
End is a no-op, matching every other Span method’s contract.

Unblocks issue 659 (ext/tasks task lifecycle linking, landed in
PR 719) and the detached edge of issue 664 (server outbound
reverse-call spans linked to the originating client request).

New-root-span marker — landed (issue 659; PR 719)

Third small contract helper, added alongside the ext/tasks consumer
work: core.WithNewRootSpan(ctx) Context and
core.IsNewRootSpanRequested(ctx) bool. Marks ctx so the next
TracerProvider.StartSpan / StartSpanLinked call produces a span with
no parent, even when ctx carries an inherited parent (a
core.TraceContext attached upstream, or the adapter’s own internal
span context, e.g. the OTel trace.SpanContext installed by a
previous StartSpan).

The motivation is async surfaces whose spanned work conceptually
outlives the originating request: a long-running task.execute
nested under a few-ms create span renders as a multi-hour child of an
already-ended parent in observability backends. Pair with
StartSpanLinked plus a Link back to the originating span so the
lifecycle stays navigable across the boundary:

bgCtx = core.WithNewRootSpan(bgCtx)
bgCtx, span := core.StartSpanLinked(tp, bgCtx, "task.execute",
    []core.Link{core.LinkFromTraceContext(originatingTC)},
    core.Attribute{Key: "mcp.task.id", Value: taskID},
)

The ext/otel adapter honors the marker by stripping any OTel span
context already on ctx before calling tracer.Start. Adapters that
don’t honor the marker (or NoopTracerProvider) silently ignore it.
The spawned span starts under whatever parent ctx happened to carry,
which degrades to the same trace tree the unmarked path would
produce. Best-effort by design.

mcpotel.NewTracerProvider helper — landed (issue 674)

Examples and surface integrations no longer have to import
go.opentelemetry.io/otel/sdk/resource + .../semconv just to set a
service.name on their TracerProvider. mcpotel.NewTracerProvider
is a thin functional-options wrapper around sdktrace.NewTracerProvider:

otelTP := mcpotel.NewTracerProvider(exp,
    mcpotel.WithServiceName("my-server"),
    mcpotel.WithSyncer(),
)
mcpotel.NewProvider(otelTP)

Options today:

  • mcpotel.WithServiceName(name) — bakes the value into an OTel
    Resource with semconv.ServiceName(...). Empty name is a no-op
    so defensive callers can pass through config values without
    branching.
  • mcpotel.WithSyncer() — switches the default Batcher to a sync
    span processor. Right for teaching demos and tests; production
    servers should stay on the batched default and handle SIGTERM with
    explicit ForceFlush + Shutdown.

NewTracerProvider(nil) panics, because silently constructing a
TracerProvider that emits no spans loses signals without surfacing
the misconfig. Matches NewProvider(nil)’s fail-fast.

Future options (WithDeploymentEnvironment, WithServiceVersion,
WithResource(*resource.Resource) escape hatch) compose against the
same internal config. File one when a consumer asks. The
examples/common/otel/ sub-package already migrated to consume this
helper; future P6 surface examples (issues 658 / 659 / 660 / 664)
inherit the cleanup without any extra plumbing.

The pattern is identical across all five: each ext/ surface optionally
accepts a core.TracerProvider and instruments its own work, depending on
the core abstraction only, never ext/otel. Same composition shape
ext/events uses for core.Claims. Noop / nil = zero overhead.

examples/common/otel.SetupTelemetry — landed (issue 666; PR 684 + PR 689)

The example layer now ships a uniform, env-gated observability
surface so every example presents the same --exporter / --otlp-endpoint
flag pair regardless of which SEP it demos. examples/common/otel/setup.go:

tp, shutdown, err := commonotel.SetupTelemetry(ctx,
    commonotel.WithExporter(*tel.Exporter),
    commonotel.WithOTLPEndpoint(*tel.OTLPEndpoint),
    commonotel.WithServiceName("my-example"),
)
defer shutdown(context.Background())
common.RunServer(common.ServerConfig{TracerProvider: tp, ...})

The EXPORTER selector is four-valued:

Value Behavior
"" (default) core.NoopTracerProvider{} + no-op shutdown. Zero overhead.
"stdout" stdouttrace exporter (sync). Demo / teaching mode.
"otlp" TCP-probe the endpoint; on success, otlptracegrpc exporter. On failure: Noop with warning log.
"auto" TCP-probe the endpoint; on success, otlptracegrpc exporter. On failure: Noop silently (operator opted into maybe-on-maybe-off semantics).

Three load-bearing details:

  • TCP probe gates OTLP. otlptracegrpc.New is lazy and returns a
    non-nil exporter even when the endpoint refuses; without the
    500ms TCP probe in probeOTLPEndpoint, the “dial-failure → Noop”
    contract wouldn’t fire (failures would surface later on first
    Export, way past just demo startup).
  • Service-name convention. Server side uses <example-name>;
    walkthrough (client) side uses <example-name>-host. Grafana’s
    service filter distinguishes the two halves of each stitched trace.
  • examples/otel/stdout/ is the documented carve-out. Its
    defaultExporter = "stdout" because the demo’s whole purpose is
    showing spans. Every other example defaults to "".

Client-side counterpart:

tel := common.ExporterFromArgs()  // os.Args scan, mirrors common.ServerURL pattern
tp, shutdown, err := commonotel.SetupClientTelemetry(ctx,
    commonotel.WithExporter(*tel.Exporter),
    commonotel.WithOTLPEndpoint(*tel.OTLPEndpoint),
    commonotel.WithServiceName("my-example-host"),
)
defer shutdown(context.Background())
c := client.NewClient(url, info, client.WithTracerProvider(tp))

SetupClientTelemetry pre-sets
WithInstrumentationName("github.com/panyam/mcpkit/client") so spans
group correctly by library in OTel backends.

PR 684 adopted across 10 example servers (auth, elicitation,
file-inputs, fine-grained-auth, list-ttl, mrtr, skills, stateless,
tasks, tasks-v2); PR 689 followed with the same uniform wiring across
9 walkthroughs. Pattern documented in examples/CONVENTIONS.md
§Telemetry wiring.

ext/auth JWT validator instrumentation — landed (issue 658; PR 694)

First P6 surface child to land on the spine. JWTConfig.TracerProvider core.TracerProvider opts the validator into instrumentation:

  • auth.jwks_lookup sub-span wraps each JWKSKeyStore.GetKeyByKid call (with mcp.auth.jwks.kid attribute; records error on lookup failure). Surfaces both cache-hit and cache-miss latency.
  • mcp.auth.* attributes on the active dispatch span (via core.SpanFromContext): mcp.auth.method = "jwt" stamped early (failure paths included), mcp.auth.subject / issuer / scopes / cache_hit set after claims extraction.

ext/auth depends on core.TracerProvider only for its own spans, with no compile-time dependency on ext/otel. Nil and core.NoopTracerProvider{} both produce zero spans, zero attributes, zero allocation. Documented in ext/auth/docs/DESIGN.md § Tracing.

oneauth v0.1.17 wiring — landed (PR 699)

Threads oneauth’s own internal spans through ext/auth so an end-to-end auth trace shows the inside of the JWKS call too. Three layered helpers enable this without abstraction leakage:

  • ext/otel.Provider.OTelTracerProvider() trace.TracerProvider — Provider stashes the underlying OTel TP and exposes it. Use when a downstream library needs the OTel SDK type directly.
  • examples/common/otel.UnderlyingOTelTP(tp core.TracerProvider) trace.TracerProvider — type-asserts to *mcpotel.Provider; returns nil for Noop or non-mcpotel providers (oneauth’s options no-op cleanly on nil).
  • ext/auth.JWTConfig.OneauthTracerProvider trace.TracerProvider — when set, threaded via keys.WithTracerProvider on the JWKSKeyStore so oneauth’s internal HTTP / parsing / signature-verify work emits spans on the same OTel pipeline.

End-to-end trace shape (when both TracerProvider AND OneauthTracerProvider are wired):

client.tools/call          (mcpkit/client)
└─ server.tools/call       (mcpkit/server, parent via _meta.traceparent)
   ├─ auth.jwks_lookup     (ext/auth)
   │  └─ oneauth.jwks.refresh
   │     └─ oneauth.jwks.key_lookup
   └─ tool handler work

Tracked in panyam/oneauth#254 (closed in oneauth v0.1.14; mcpkit ext/auth on v0.1.17 after a CI fix that swept all 8 oneauth-using modules in lock-step). Adopter pattern shipped in examples/auth/common/setup.go via WithMCPTracerProvider / WithOneauthTracerProvider variadic options on Env.NewValidator. Demo wires both in examples/auth/main.go.

Apps Bridge trace context relay — landed (issue 660; PR 702)

Bidirectional W3C trace context propagation across the iframe ↔ host postMessage boundary so a browser-side trace (browser OTel SDK / RUM / hard-coded demo traceparent) stitches with the backend tool-call span. Off by default on both sides; adopters opt in independently per side.

TS bridge (ext/ui/assets/mcp-app-bridge.ts):

MCPApp.setTraceContextProvider(() => ({
  traceparent: "00-...-...-01",
  tracestate: "vendor=val",  // optional
}));

The provider is called before every outbound request() / notify(). Result merges into params._meta.traceparent / _meta.tracestate. Caller-set _meta wins (provider is a fallback). Provider throws are caught and logged; the request still sends without propagation. dep-free, with no OTel JS dependency.

Go AppHost (ext/ui/app_host.go):

host := ui.NewAppHost(c, bridge, ui.WithTracerProvider(tp))

handleAppRequest extracts inbound params._meta.traceparent via core.ExtractTraceContextFromParams, wraps the forward to the MCP server in an apps.host.forward span parented by the iframe’s traceparent. The outbound client.Call preserves the iframe’s _meta.traceparent on the wire (SEP-414 P3’s caller-set-wins contract) so the server’s P2 dispatch span stitches as a child.

End-to-end trace shape:

iframe traceparent          (browser-side OTel; demo: hard-coded random)
└─ apps.host.forward        (ext/ui AppHost)
   └─ server tools/call     (mcpkit server, SEP-414 P2)
      └─ tool handler work

Demo wiring in examples/apps/vanilla/dice.html (per-page-load random traceparent + window.__MCPAPP_DEMO_TRACEPARENT__ for on-page TraceID copy). Host wiring in examples/host/01-apphost/main.go. Design doc: docs/APPS_DESIGN.md § Tracing across the Apps Bridge.

Open spec question: the Apps Bridge is a non-MCP transport; whether the relay belongs in the Apps spec for cross-SDK interop is open. mcpkit ships the relay; upstream note filed only if working-group interest surfaces.

Events bus trace context relay — landed (issue 683; PR 712 + PR 714)

W3C Trace Context propagates across every gate in the events lifecycle so a yield on replica A and a downstream webhook delivery (or a poll-side replay on replica B) appear in Tempo as one stitched trace. Sister to the Apps Bridge relay, the same “non-MCP propagation hop” pattern across a different boundary topology.

Four gates, three carriers, one consistent rule. Each gate picks the carrier appropriate to its boundary:

Gate Carrier Where
1. yield(ctx, data) event.Meta.traceparent (persistent) + ctx (in-process) YieldingSource.yield stamps Meta from ctx; emit hook receives the same ctx
2. emit hook → Emitter.Emit(ctx, event) ctx Register passes the hook’s ctx straight to the configured Emitter
3. WebhookRegistry.Deliver(ctx, event) → outbound HTTP HTTP traceparent header deliver() extracts from ctx (preferred) or event.Meta (fallback for replayed events), stamps the header before client.Do
4. HTTPSource.serveInject (receiving replica) inbound HTTP header → ctx → event.Meta Handler reads the traceparent header, builds core.TraceContext, attaches via core.WithTraceContext, calls s.yield(ctx, data) — closes the round-trip

Caller-preserves rule. If SetMetaFunc pre-stamps event.Meta.traceparent, the yield-time auto-injection is skipped, uniform with core.InjectTraceContextIntoParams and the TS-side Apps Bridge relay.

events.webhook.deliver span (PR 714). WebhookRegistry.WithWebhookTracerProvider(tp) opts the registry into emitting a span around each retry loop. Attributes: webhook.target.id, webhook.url, mcp.event.name, http.method, http.response.status_code, webhook.retry.attempts. RecordError fires on retries-exhausted with the categorical bucket. Nil = Noop, zero overhead. Live-verified against the LGTM stack: span landed with all attributes set, 4 attempts counted, STATUS_CODE_ERROR on the failure path, duration matched the 0.5s + 1s + 2s backoff schedule exactly.

Server.Broadcast(ctx, ...) (PR 714 signature; PR 722 body). PR 714 widened the signature; PR 722 (issue 715) made the body actually consume ctx. When ctx carries a non-zero core.TraceContext, the inbound traceparent / tracestate are injected into notification params under _meta (via core.InjectTraceContextIntoParams) before fan-out. Existing caller-set _meta.traceparent wins. Per-transport broadcast callbacks gained ctx as a forward-compat seam, unused inside today because the per-request trace_middleware.go wrap (core.WrapSessionNotifyFunc) targets sc.notify (the handler-facing notify on a per-request SessionCtx), not the transport-level base notifyFunc dispatchers expose to broadcasts. Injecting once at the Server level keeps tracing concerns out of the transport-level loops while still stitching SSE-pushed notifications into the originating trace.

End-to-end trace shape (multi-replica with webhook fanout):

yield(ctx, data) on replica A
└─ event.Meta.traceparent stamped
   └─ emit hook (ctx) → LocalEmitter → WebhookRegistry.Deliver(ctx, event)
      └─ events.webhook.deliver span (PR 714)
         └─ HTTP POST to replica B with traceparent header
            └─ replica B HTTPSource.serveInject reads header
               └─ yield(ctx, data) on replica B (same TraceID)

Demo wiring lives in examples/whole-enchilada/events/event-server/main.go (events.WithWebhookTracerProvider(tp) threaded against the existing commonotel.SetupTelemetry TP). Adopter sweep (PR 714) updated examples/events/discord and examples/events/telegram to the new yield(ctx, ...) / SetMetaFunc(ctx, ...) signatures. Cross-replica peer-fanout Emitter (Redis pubsub) tracked on panyam/mcpkit#634 + panyam/mcpkit#639 with concrete design comments added.

Events fanout span emission — landed (issue 724; PR 730)

Companion to the bus-relay work above. While 683/712/714 handled propagation (trace context flowing across the lifecycle gates), 724 handles emission on the per-yield fanout itself. Adopters opt in via events.Config.TracerProvider core.TracerProvider. Register threads it onto every Source implementing the new TracerProviderInstaller interface (YieldingSource does; TypedSource authors can implement it identically).

events.Register(events.Config{
    Server:         srv,
    Sources:        []events.EventSource{chatSrc, alertSrc},
    Webhooks:       webhooks,
    TracerProvider: tp, // ← new in PR 730
})

Span shape: one events.fanout span per emitted event (NOT per subscriber), parented by the yield ctx. Attributes:

Attribute Meaning
mcp.event.name the EventDef.Name
mcp.event.id the assigned EventID (cross-referencing)
events.subscribers.total snapshot subscriber count at fanout time
events.subscribers.delivered count that passed Match
events.subscribers.dropped_by_match count where Match returned false
events.transforms.applied count where Transform actually modified the event

Design call locked in: one span per yield, NOT per subscriber. Option (a) from the original issue body (events.match / events.transform per Match/Transform invocation) was deliberately NOT shipped, since it would scale linearly with subs × events and need sampling design. The aggregate counts are the diagnosable shape; operators see “this yield went to 10 subs, 7 dropped by Match” without span-volume risk. If per-subscriber detail ever becomes load-bearing, option (a) can land as a separate opt-in on the same TracerProvider seam.

Two load-bearing optimizations on the hot path:

  • Zero-subscriber guard. len(subs) == 0 skips span emission entirely. Idle sources (feeders with no subscribers registered yet) are dominant, and emitting empty fanout spans every feeder tick would flood Tempo with noise.
  • Noop short-circuit. tp.(core.NoopTracerProvider) type-assertion avoids the StartSpan call entirely on the unconfigured path. Per-yield cost on the unconfigured path is one type-assertion.

NOT in scope (deferred to future PRs if anyone asks): webhook-side Match/Transform attributes on the existing events.webhook.deliver span (PR 714); poll-side Match/Transform aggregate attributes on the events/poll dispatch span. Both are covered by existing spans; decorating with more attributes is drive-by.

Test pattern: reuses the fakeTP recording machinery from webhook_span_test.go so the assertion style stays consistent across event-surface span tests. Tests cover attribute shape, drop-by-Match, transforms-applied, zero-subscriber skip, parent stitching, Noop overhead, and the Register install path.

End-to-end trace shape:

tools/call (dispatch, SEP-414 P2)
└─ events.fanout (PR 730 — one span per yield, parented here)
   ├─ events.subscribers.total = 10
   ├─ events.subscribers.delivered = 3
   ├─ events.subscribers.dropped_by_match = 7
   └─ events.transforms.applied = 3
   └─ events.webhook.deliver (PR 714 — per webhook target, sibling span via existing seam)

Doc: experimental/ext/events/README.md § Tracing across the events bus + examples/events/kitchen-sink/README.md § Tracing.

ext/tasks task lifecycle spans — landed (issue 659; PR 719)

Final P6 surface child. A tools/call that creates a task spawns a background goroutine whose work outlives the create span. Parent-child doesn’t fit, so the tasks runtime uses span links instead. The dispatch span for the tools/call ends in a few ms; task.execute runs as a NEW root trace (potentially for hours) with a Link back to the create span. Every later tasks/get / tasks/update / tasks/cancel dispatch span gets an AddLink back to the same create span so a backend can pivot from any poll into the whole lifecycle.

Wiring: tasks.Config.TracerProvider core.TracerProvider. Nil or core.NoopTracerProvider{} (the default) skips the install, so there is zero overhead, no spans, and no allocation. ext/tasks depends on the core abstraction only; no compile-time dep on ext/otel.

tasks.Register(tasks.Config{
    Server:         srv,
    TracerProvider: mcpotel.NewProvider(otelTP),
})

Span shape:

Span Lifecycle Parent Links Attributes
tools/call (dispatch) request scope inbound _meta.traceparent — SEP-414 P2 dispatch attrs
task.execute goroutine lifetime none (new root via WithNewRootSpan) → tools/call create span mcp.task.id, mcp.task.status (stamped at End)
tasks/get / tasks/update / tasks/cancel request scope inbound _meta.traceparent → tools/call create span (AddLink) SEP-414 P2 dispatch attrs

Status stamping: the goroutine’s defer reads the final status from the store and stamps mcp.task.status (completed / failed / cancelled / input_required) on the span, then Ends. SEP-2663 semantics: handler-returned errors ((nil, err)) map to completed with IsError=true on the inner ToolResult. RecordError on the span is reserved for protocol-level failures (mwErr, resp.Error, unexpected result shape, panic recover).

Sync-as-completed path is deliberately silent. wrapSyncAsCompletedTask (a sync-returning handler under TaskSupport=optional/required) runs inside the create span, and emitting a zero-duration task.execute for the wire-uniformity wrapper would be noise. Only the GoAsync path (the work that actually escapes the request span) gets task.execute.

Lifetime of originTC (the runtime map that stashes the create-span TraceContext per taskID): outlives the activeTask entry. Polls landing on a terminal task still get the AddLink back to the create span. Matches the existing pattern for taskErrors / creatorToolForTask.

End-to-end trace shape:

tools/call (dispatch)                         (mcpkit/server, SEP-414 P2)
└─ [link] task.execute  ────────────────►     (root trace, ext/tasks goroutine, mcp.task.id, mcp.task.status)
                                              (tool handler work runs inside task.execute)

tasks/get (dispatch, link → original tools/call create span)
tasks/update (dispatch, link → original tools/call create span)
tasks/cancel (dispatch, link → original tools/call create span)

End-to-end materialization is proven by ext/otel/provider_test.go’s coverage of the WithNewRootSpan scrub + StartSpanLinked + AddLink paths against a real OTel SDK exporter. ext/tasks/trace_test.go uses a recording fake core.TracerProvider + core.LinkedTracerProvider to prove ext/tasks calls the contract correctly without dragging ext/otel into its module graph.

MRTR multi-round trace stitching — landed (issue 682; PR forthcoming)

SEP-2322 MRTR splits one logical operation across N JSON-RPC dispatches (round 1: server returns InputRequiredResult → client gathers input → round 2: client retries with inputResponses). Each round’s tools/call mints its own span on each side, so without intervention an MRTR operation produces N unrelated traces, so an operator looking at the final ToolResult’s trace cannot navigate to the input-gathering work that produced it.

This is fundamental once stateless (SEP-2575) replaces stateful as the default wire. Stateful’s ctx.Sample / ctx.Elicit are deprecated per SEP-2577 and will go away in v0.4. MRTR becomes the canonical input-gathering pattern; the trace story has to follow.

Wire shape: rounds 2+ carry the round-1 traceparent in a new _meta.io.modelcontextprotocol/tracelink field. The server’s trace middleware reads it and calls core.SpanFromContext(ctx).AddLink(...) on the round-N dispatch span, stitching the operation across separate W3C traces.

Round 1:                              Round 2 (and beyond):
client.Call("tools/call", args)       client.Call("tools/call", args+inputResponses)
  ↓                                     ↓
_meta: {                              _meta: {
  traceparent: T1                       traceparent: T2,
}                                       io.modelcontextprotocol/tracelink: T1,  ← NEW (issue 682)
                                      }
  ↓                                     ↓
server span (TraceID T1)              server span (TraceID T2)
                                        AddLink → T1  ← server stitches the operation

The star semantic means rounds 2, 3, 4… all link back to round 1, NOT the immediately-previous round. Backends show round 1 as the anchor regardless of how deep the MRTR loop goes. One click navigates from any round to the originating request.

Vendor-namespaced field: _meta.io.modelcontextprotocol/tracelink (not bare _meta.tracelink) because W3C doesn’t define a tracelink field, and bare names are reserved for W3C-defined keys. mcpkit-specific until upstream WG agrees on a cross-SDK shape.

Client-side identity capture: CallToolWithInputs uses the new core.WithCapturedTraceContext ctx-holder primitive. The client trace middleware writes round-1’s outbound TraceContext into the holder; CallToolWithInputs snapshots it and stamps it onto every subsequent round’s params via core.InjectTraceLinkIntoParams. Tracestate is intentionally NOT part of the link payload, since the link is identity-only rather than full propagation context. Client.callImpl(ctx, ...) (a ctx-aware refactor of Client.Call) is what threads the holder through the middleware chain; Client.Call itself stays ctx-free for back-compat by passing context.Background().

Considered alternatives:

  • Embed traceparent in requestState — works, but requestState was designed as opaque from the client’s perspective. Baking tracing semantics into it muddies the contract.
  • Continue an outer span across rounds on the client side — simpler (no wire change), but the outer span’s duration would include user-paced gather-input time, which can be minutes. Distorts span duration semantics. Rejected.

Spec engagement TBD: this lands as mcpkit-internal first; the field name + semantic could be a candidate for upstream MCP WG standardization for cross-SDK interop. See the PR description for the working-group post draft.

End-to-end correctness proof: ext/otel/mrtr_tracelink_e2e_test.go drives a real CallToolWithInputs against a real OTel-SDK-backed server + client, then asserts the recorded round-2 server span has an OTel Link whose TraceID matches round-1’s.

Skills observability — landed (issue 748; PR forthcoming)

Tool calls have natural telemetry: every tools/call is one wire event → one server-side dispatch span. Skills (SEP-2640) don’t. The host fetches a manifest once via resources/read, caches it client-side, then activates it N times invisibly. Without intervention, post-cache activation is unobservable to either side.

Issue 748 closes the gap in two layers.

Layer 1 — server dispatch span enrichment. server/trace_middleware.go now stamps the resources/read span with mcp.resource.uri (always, for any URI) plus mcp.skill.uri / mcp.skill.path / mcp.skill.file when the URI uses the skill:// scheme. Mirrors the existing tools/call → mcp.tool.name pattern. Server-side dashboards can now chart fetch volume + error rate per skill.

mcp.skill.path and mcp.skill.file only populate for SEP-2640 manifest URIs (terminal /SKILL.md) where the path/file boundary is unambiguous from the URI alone. Non-manifest URIs (e.g. skill://pdf-processing/references/FORMS.md) surface as mcp.skill.uri only. SEP-2640 documents that the boundary in those cases requires external knowledge from the discovery index or a prior manifest read.

Layer 2 — client-side ext/skills.Client instrumentation. NewClient(mcp, opts...) is variadic; new WithTracerProvider(tp) option wraps the read-path methods in spans:

Method Span name Attributes
ListSkills skills.list mcp.skill.uri (index URI), mcp.skill.count on success
ReadSkillURI skills.read mcp.skill.uri
ReadSkillManifest skills.read_manifest mcp.skill.uri, mcp.skill.path, mcp.skill.name
ReadAndVerify skills.read_and_verify mcp.skill.uri, mcp.skill.expected_digest, mcp.skill.digest_verified

Stitching is automatic: the client read span runs inside ctx that already carries _meta.traceparent from the existing W3C trace context propagation (PR 644 / PR 649 / PR 652), so the server-side dispatch span (now enriched per Layer 1) lands as a child of the client wrapping span in the same trace.

The activation hook — purely SDK-side, no wire change. Client.Activate(ctx, uri, opts...) ActivationEvent is the answer to the “skill used” telemetry the wire can’t capture. Hosts call it at the point in the agent loop where the skill enters model context. Side effects:

  • Emits an instant skills.activate span (Start + End back-to-back, since activation is point-in-time rather than a duration) carrying mcp.skill.uri, mcp.skill.path (when the URI parses as a manifest URI), and mcp.skill.activation.reason when WithReason(...) is supplied.
  • Invokes the optional WithActivationHook(fn) callback synchronously, so non-OTel hosts can feed their own telemetry pipeline (structured logging, internal counters, alternate tracing) without configuring a TracerProvider.
  • Returns the ActivationEvent{URI, Reason, Timestamp} to the caller for pull-style stamping.

Activate accepts any string URI, and it never blocks dispatch or validates input. Designed for the agent-loop case where the URI may be a manifest URI, a sub-file URI, or even an opaque identifier the host minted for an in-process bundle.

Considered alternatives:

  • Wire-level notifications/...skills/activated — a cross-process notification carrying {uri, digest, reason, _meta.traceparent} so the server can record activation events from connected clients. Tractable but cross-cuts SEP-2640 spec territory and requires WG engagement before minting the method name. Tracked as issue 749, a separable opt-in extension on ext/skills that can ride atop Client.Activate without breaking callers when (if) the upstream method name lands.
  • Provider-side activation telemetry — N/A. The provider handles resource reads; activation is a client/host concept.
  • Activation counter on the issue 735 MeterProvider seam — useful follow-up. The TracerProvider half ships in 748; counter wiring can land cleanly in a small follow-up PR.

Coverage:

  • server/trace_middleware_test.go — three tests pin Layer 1: skill manifest URI emits all four attrs; non-skill URI emits mcp.resource.uri only; non-manifest skill URI emits mcp.skill.uri only.
  • ext/skills/client_trace_test.go — eight tests pin Layer 2: each wrapped method emits the right span + attrs; Activate fires hook + span; Activate omits the reason attr when not supplied; hook fires independently of tracer install; zero-overhead path (no WithTracerProvider) emits no observable spans.
  • End-to-end trace context propagation across resources/read is the same code path the MRTR e2e already exercises for tools/call (_meta.traceparent injection in dispatch is method-agnostic). A skills-specific real-SDK e2e is deliberately omitted, since it would require either inverting the ext/otel-doesn’t-import-ext/skills layering or adding the real OTel SDK as a test-only dep on ext/skills.

Runnable demo: examples/skills/walkthrough.go exercises the wrap span + Activate path with the --exporter=stdout (or --exporter=otlp) flag pair. Run with EXPORTER=stdout just serve + EXPORTER=stdout just demo.

experimental/ext/agents discovery spans — landed (issue 1145)

The server-declared agents primitive (experimental/ext/agents, epic 1142) advertises a roster via agents/list and resolves a specialist’s instructions plus scoped tools via agents/get. The WG frames MCP’s subagent gap versus ACP as an observability gap, so tracing the discovery + delegation lifecycle is part of the primitive’s value, not a parity afterthought.

The pass follows the established opt-in contract. agents.Config.TracerProvider core.TracerProvider defaults to core.NoopTracerProvider{} (zero allocation, no spans), and the extension depends only on the core tracing abstraction, never on ext/otel.

Server side — discovery spans (experimental/ext/agents/registry.go):

Handler Span name Attributes
agents/list agents.list agents.count (roster size)
agents/get agents.get mcp.agent.id, agents.found (true / false)

agents.found is set on every path (including empty / unknown agentId), so a failed lookup is visible in the trace rather than silent.

Host side — delegation edge (chakra’s host/server_agents.go). The host’s lazy serverAgentSource.resolve wraps the first-use agents/get + child build in an agents.resolve span carrying mcp.agent.id. Cache hits are not spanned. This ties the three pieces a delegation trace should show: supervisor turn -> agents.resolve(agent.id) -> agents.get -> child turn.

Sub-agent execution reuses the Runner’s own spans. A resolved specialist is a normal agent.AgentSource over a child Runner, so its work already emits agent.turn / agent.step / agent.tool spans (the same instrumentation any Runner gets). The host threads its TracerProvider through the bridge (ServerAgentConfig.TracerProvider), so no separate execution-span machinery is needed. The in-process delegation runs synchronously inside the supervisor’s tool-call span, so those child spans nest naturally under it. A new-root-span + link shape (like task.execute) would only be warranted for an async delegation form (AsyncAgentSource) whose run outlives the dispatch; that is deferred until a demo needs it.

Coverage: experimental/ext/agents/tracing_test.go pins the discovery spans (list count, get found / not-found, and the zero-overhead nil-TracerProvider path); chakra’s host/server_agents_test.go pins the agents.resolve span (emitted on first delegation, not on the cached second).

Downstream consumers of the Phase 1 contract

  • experimental/ext/events/ cross-replica bus (issue #629). The
    EventBus envelope will carry traceparent / tracestate using the
    core.MetaKey* constants, so a trace started on replica A and
    delivered from replica B stitches together once an OTel adapter is
    wired. The seam consumes core.TracerProvider directly, with no
    ext/otel import in the base events module, mirroring how events
    already consumes core.Claims without depending on ext/auth.
  • server/ topology preflight (issue #642). The capability
    contract declares whether a configured seam supports trace
    propagation; the declaration uses the core.TracerProvider
    interface to keep the contract dep-free.