mcpkit

Observability

SEP-414 OpenTelemetry tracing across the server, client, tasks, events, and the Apps bridge.

Source: docs/SEP_414_OTEL.md

SEP-414: OpenTelemetry Trace Context Propagation

Status: Phases 1, 2, 3, and 4 landed. Polished walkthroughs (Phase 5)
and the conformance suite (issue 429) still to come.

This document is the wiring guide for mcpkit’s OpenTelemetry surface. It
covers what the shipped phases give you, what to expect once you wire a
real TracerProvider, and where the remaining work lives on
issue 312.

What Phase 1 ships

Phase 1 lands the dependency-free contract surface in core/:

  • core.TracerProvider, core.Span, core.Attribute — the minimal
    tracing seam mcpkit components consume. The default,
    core.NoopTracerProvider, performs no allocation and emits no spans.
  • core.TraceContext — the W3C traceparent / tracestate pair as
    propagated on the MCP wire.
  • core.ExtractTraceContext / core.InjectTraceContext — read/write
    W3C Trace Context from/to an MCP _meta map, with strict
    W3C-version-00 validation on extract.
  • core.WithTraceContext / core.TraceContextFromContext —
    context.Context plumbing.
  • core.BaseContext.TraceContext() — accessor exposed on every typed
    handler context (ToolContext, PromptContext, ResourceContext,
    MethodContext) so handlers can read the active trace context
    without import gymnastics.

The MetaKeyTraceparent / MetaKeyTracestate constants pin the wire
keys (bare traceparent and tracestate, matching W3C HTTP header
names — not under the io.modelcontextprotocol/ namespace, which is
reserved for MCP-defined fields).

Phase 1 introduces no behavior change: nothing in mcpkit calls
TracerProvider.StartSpan yet, and no transport reads or writes the
_meta trace keys. The surface exists so downstream work can be
written and reviewed against a stable contract.

What Phase 2 ships

Phase 2 wires the server-side propagation surface:

  • server.WithTracerProvider(tp core.TracerProvider) Option — install a
    tracer. Default and core.NoopTracerProvider{} both skip the trace
    middleware entirely so the zero-overhead path stays untouched.
  • An internal traceMiddleware wraps every JSON-RPC dispatch in an
    inbound span, positioned OUTERMOST so user-registered middleware
    contributes to the recorded latency. Span attributes:
    • mcp.method on every span
    • mcp.tool.name on tools/call
    • mcp.session.id when a session is attached
    • mcp.error.code + Span.RecordError on JSON-RPC errors
    • mcp.tool.is_error="true" on tools/call results carrying IsError
  • Inbound trace context resolution: params._meta.traceparent wins
    over the out-of-band TraceContext bridged through context.Context.
    Malformed traceparent values are dropped per the W3C “MUST NOT
    forward” rule — the span still emits with no parent.
  • The streamable HTTP transport bridges the W3C traceparent /
    tracestate HTTP headers (SEP-2028) into context.Context once at
    the POST entry point. All downstream dispatch paths (sync JSON, SSE,
    batch, stateless, initialize) inherit the bridge.
  • Outbound _meta.traceparent / _meta.tracestate are injected on
    every server-to-client message — notifications (logging, progress,
    resource updates, custom) and server-to-client requests (legacy push
    for sampling/elicitation/roots). The wrap sits OUTSIDE any
    user-registered NotifyInterceptor / RequestInterceptor, so those
    interceptors observe the same wire form the client receives.

Wiring is a single line in your server bootstrap:

srv := server.NewServer(info,
    // ...your other options...
    server.WithTracerProvider(myTracerProvider),
)

myTracerProvider must implement core.TracerProvider. Until the
Phase 4 ext/otel/ adapter ships, the only options that emit spans are
custom implementations — the in-tree core.NoopTracerProvider
intentionally treats “spans” as no-ops.

What Phase 4 ships

Phase 4 lands the OpenTelemetry SDK adapter as a new sub-module under
ext/otel/ (separate go.mod, mirroring ext/auth/ / ext/tasks/ /
ext/ui/). Wire it in with one line:

import (
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"

    "github.com/panyam/mcpkit/server"
    mcpotel "github.com/panyam/mcpkit/ext/otel"
)

exp, _ := stdouttrace.New()
otelTP := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp))
defer otelTP.Shutdown(ctx)

srv := server.NewServer(info,
    server.WithTracerProvider(mcpotel.NewProvider(otelTP)),
)

Behavior:

  • The adapter parses core.TraceContext from ctx into an OTel
    SpanContext so the inbound parent flows into the SDK’s standard
    parent-tracking machinery.
  • After StartSpan, it formats the new span’s SpanContext as a W3C
    traceparent and re-attaches via core.WithTraceContext, so the
    Phase 2 outbound _meta injection wraps stamp every server-to-client
    message with the child span’s traceparent (not the inbound
    parent’s).
  • core.Attribute{Key, Value string} maps to attribute.String(...);
    RecordError(err) emits an OTel “exception” event AND sets the span
    status to codes.Error; End() is idempotent.
  • WithInstrumentationName(...) overrides the default
    instrumentation library name ("github.com/panyam/mcpkit/server").

Verification:

  • just test-otel — adapter unit tests run against the real OTel SDK
    in-memory exporter; reads back sdktrace.ReadOnlySpan shapes.
  • just test-otel-example — smoke test for examples/otel/stdout/
    asserting the exporter actually prints the expected span set.
  • go run examples/otel/stdout/... — runnable demo, prints spans as
    JSON on stdout. No collector required.

See ext/otel/README.md for the API reference
and examples/otel/stdout/README.md
for the walkthrough.

What Phase 3 ships

Phase 3 lands the symmetric client-side surface:

  • client.WithTracerProvider(tp core.TracerProvider) — install a
    tracer. Default and core.NoopTracerProvider{} both skip the install
    (zero overhead on the unconfigured path).
  • Outbound Client.Call is wrapped in a span via a ClientMiddleware
    installed as the OUTERMOST entry — user middleware (auth retry,
    header injection) runs inside the span so its latency is captured.
    Span attributes:
    • mcp.method on every span
    • mcp.tool.name on tools/call
    • mcp.client.session.id when a session exists (note client.
      namespace — operators want to filter by client- vs
      server-emitted)
    • mcp.error.code + Span.RecordError on *RPCError
  • Outbound params gain _meta.traceparent / _meta.tracestate
    derived from the active span via core.InjectTraceContextIntoParams
    (promoted from the server-internal helper so both wires apply the
    same precedence rule — explicit caller-set values win).
  • Inbound server-to-client requests
    (sampling/createMessage, elicitation/create, roots/list):
    params._meta.traceparent is extracted via
    core.ExtractTraceContextFromParams, attached to ctx via
    core.WithTraceContext, and a wrap span named after the request
    method is emitted with the inbound trace context as parent. The
    handler observes the inbound TC via core.TraceContextFromContext(ctx)
    and can use it as a parent for its own spans.

Wiring is a single line in your client bootstrap:

c := client.NewClient(url, info,
    // ...your other options...
    client.WithTracerProvider(myTracerProvider),
)

myTracerProvider is the same core.TracerProvider you’d hand the
server side via server.WithTracerProvider. The Phase 4 ext/otel/
adapter (PR 652) is the in-tree implementation.

What comes next

Tracked phases on issue 312:

  • P5 — polished examples. examples/otel/jaeger/ +
    examples/otel/otlp/ end-to-end walkthroughs with collector and UI
    screenshots. The minimal examples/otel/stdout/ already covers
    smoke-verification.
  • Conformance suite testconf-otel — issue 429.
  • P6 — tracing across surfaces. Umbrella issue 663. See below.

Adjacent: W3C Baggage propagation + HTTP forward helper (issue 739)

W3C Baggage is a separate W3C standard from W3C Trace Context
(traceparent / tracestate) that propagates arbitrary key=value
pairs alongside trace identity — operators typically use it for
tenant_id, user_id, feature flags, A/B-test buckets. mcpkit
propagates it symmetrically to the trace context surface so handlers
can ctx.Baggage() and stamp it onto outbound work without
re-implementing the W3C parsing rules.

core.HTTPForwardTransport(base http.RoundTripper) wraps an
http.Client’s Transport so a tool handler’s downstream HTTP
calls automatically carry the active traceparent / tracestate /
baggage headers. This closes the end-to-end observability loop for
handlers that call third-party APIs — without it, the trace stops at
the MCP boundary and resumes on the downstream side as a fresh
trace.

These features ship independently of upstream SEP-2028 (which
adds a configurable headerGroups API on top of this same
foundation). Tracking the upstream spec on issue 739; the W3C
standards themselves are stable and ship today.

Adjacent: metrics seam (issue 7)

The core.MeterProvider seam mirrors the SEP-414 core.TracerProvider
shape for metrics. Metrics don’t cross the wire, so no SEP — but the
library still needs a dependency-free seam so the base module can emit
measurements without dragging the OTel metrics SDK in. Wire it via
server.WithMeterProvider; the OTel adapter lives at
mcpotel.NewMeterProvider(otelMP). Canonical instruments emitted from
the server dispatch path (mcp.tool.calls, mcp.jsonrpc.errors,
mcp.tool.duration, mcp.sessions.active) share the same attribute
vocabulary as the SEP-414 spans, and exemplars are wired by default so
Grafana’s metric → trace pivot works without per-call configuration.
See ext/otel/README.md § Metrics for the wiring snippet and
server/metrics_middleware.go for the dispatch-side instrumentation.

P6 — tracing across surfaces (auth / tasks / apps)

P1–P5 instrumented the dispatch spine: every JSON-RPC method gets one
inbound span plus W3C wire propagation, for free. P6 extends tracing into
the work each surface does that the single dispatch span doesn’t cover.

Is this a SEP-414 gap? No. Separate three layers:

Layer Owner Status
Wire contract (the _meta keys, format, precedence, SEP-2028 bridge) SEP-414 — normative, for cross-SDK interop complete
Local span richness (count, names, attributes, links) the SDK — latitude; invisible to the peer mcpkit-completeness work (P6)
Adjacent-transport hops (events bus, apps Bridge) mostly the SDK; the Apps Bridge may belong in the Apps spec open

Span richness never crosses the wire, so a SEP can’t (and shouldn’t)
mandate it. The one genuine spec question is the apps Bridge
(postMessage): if cross-SDK app-tracing interop is a goal, whether
traceparent rides the bridge envelope belongs in the ext-ui / Apps
spec, not SEP-414.

Two categories of “work the spine misses”:

  • (a) escapes the span — async / out-of-band on a non-MCP transport:
    tasks background execution (#659), events cross-replica bus
    (#642 / #629).
  • (b) inside the span, not broken out / enriched — auth validation
    sub-spans + principal attributes (#658).

The two categories map cleanly onto the contract:

  • New propagation surfaces need no new contract — core.ExtractTraceContext
    / InjectTraceContext already operate on any map[string]any, so the
    apps Bridge (#660) and the events bus reuse them verbatim.
  • Enrichment surfaces reveal the only two P1 gaps, neither of which the
    spine needed: core.SpanFromContext (active-span accessor, #661;
    unblocks auth attributes) and span links on core.Span (#662;
    unblocks task lifecycle).

Active-span accessor — landed (issue 661)

The first contract gap is closed: core.SpanFromContext(ctx) core.Span
returns the currently-active mcpkit Span (or a no-op Span when none is
attached). Sibling helper core.WithActiveSpan(ctx, span) Context is what
TracerProvider adapters call after StartSpan to publish the span. The
in-tree NoopTracerProvider and the ext/otel adapter both follow the
pattern, so the contract holds regardless of which provider is wired.

The intended use is decorating the dispatch span, not nesting a child:

// inside a middleware or handler, no ext/otel import needed:
span := core.SpanFromContext(ctx)
span.SetAttribute("mcp.auth.principal", claims.Subject)
span.SetAttribute("mcp.auth.method", "jwt")

Typed handler contexts gain a Span() accessor that delegates to
SpanFromContext:

func myTool(ctx core.ToolContext, req core.ToolRequest) (core.ToolResponse, error) {
    ctx.Span().SetAttribute("mcp.tool.cache.hit", "true")
    // ...
}

The accessor never returns nil — call sites that always-decorate work
correctly whether or not a TracerProvider is configured. Issue 658
(ext/auth attributes) and any future “enrich the active span” use case
consume this surface without crossing the core/-only boundary.

Span links — landed (issue 662)

The second contract gap is closed: span links are now expressible
dep-free, with the OTel-aligned shape that includes per-link
attributes. Two entry points:

// Option at start (call site knows all links upfront):
links := []core.Link{
    {TraceContext: tc1, Attributes: []core.Attribute{{Key: "link.kind", Value: "originated-from"}}},
    core.LinkFromTraceContext(tc2),
}
ctx, span := core.StartSpanLinked(tp, ctx, "task.execute", links, attrs...)

// Mid-flight (links discovered during span execution):
span := core.SpanFromContext(ctx)
span.AddLink(core.Link{TraceContext: tc, Attributes: ...})

core.Link mirrors the OpenTelemetry spec Link definition: a
TraceContext (the upstream identity, same shape carried on the MCP
wire via _meta) plus optional per-link attributes. Per-link
attributes are how observability backends render the link UI —
Jaeger / Tempo / Honeycomb all show link.kind=... semantically
rather than as generic span attributes.

Capability widening pattern: core.LinkedTracerProvider is a
sibling of TracerProvider that adds the StartSpanLinked method;
the core.StartSpanLinked(tp, ...) package-level helper type-asserts
and falls back to plain StartSpan when the provider doesn’t
implement the wider interface — links silently dropped, span emitted.
This kept the base TracerProvider interface unchanged so non-tracing
test fakes and the default NoopTracerProvider didn’t need any
churn. The ext/otel adapter implements both interfaces and maps
each core.Link to an OTel trace.Link via the existing
traceContextToSpanContext helper.

Invalid links are silently dropped — both at StartSpanLinked
(filtered before passing to the OTel SDK) and at Span.AddLink
(short-circuited inside the wrapper). Defensive call sites can build
link slices from raw inputs (e.g., core.ExtractTraceContext outputs
that might be zero) without pre-filtering. Calling AddLink after
End is a no-op, matching every other Span method’s contract.

Unblocks issue 659 (ext/tasks task lifecycle linking — landed in
PR 719) and the detached edge of issue 664 (server outbound
reverse-call spans linked to the originating client request).

New-root-span marker — landed (issue 659; PR 719)

Third small contract helper, added alongside the ext/tasks consumer
work: core.WithNewRootSpan(ctx) Context and
core.IsNewRootSpanRequested(ctx) bool. Marks ctx so the next
TracerProvider.StartSpan / StartSpanLinked call produces a span with
no parent — even when ctx carries an inherited parent (a
core.TraceContext attached upstream, or the adapter’s own internal
span context, e.g. the OTel trace.SpanContext installed by a
previous StartSpan).

The motivation is async surfaces whose spanned work conceptually
outlives the originating request: a long-running task.execute
nested under a few-ms create span renders as a multi-hour child of an
already-ended parent in observability backends. Pair with
StartSpanLinked plus a Link back to the originating span so the
lifecycle stays navigable across the boundary:

bgCtx = core.WithNewRootSpan(bgCtx)
bgCtx, span := core.StartSpanLinked(tp, bgCtx, "task.execute",
    []core.Link{core.LinkFromTraceContext(originatingTC)},
    core.Attribute{Key: "mcp.task.id", Value: taskID},
)

The ext/otel adapter honors the marker by stripping any OTel span
context already on ctx before calling tracer.Start. Adapters that
don’t honor the marker (or NoopTracerProvider) silently ignore it —
the spawned span starts under whatever parent ctx happened to carry,
which degrades to the same trace tree the unmarked path would
produce. Best-effort by design.

mcpotel.NewTracerProvider helper — landed (issue 674)

Examples and surface integrations no longer have to import
go.opentelemetry.io/otel/sdk/resource + .../semconv just to set a
service.name on their TracerProvider. mcpotel.NewTracerProvider
is a thin functional-options wrapper around sdktrace.NewTracerProvider:

otelTP := mcpotel.NewTracerProvider(exp,
    mcpotel.WithServiceName("my-server"),
    mcpotel.WithSyncer(),
)
mcpotel.NewProvider(otelTP)

Options today:

  • mcpotel.WithServiceName(name) — bakes the value into an OTel
    Resource with semconv.ServiceName(...). Empty name is a no-op
    so defensive callers can pass through config values without
    branching.
  • mcpotel.WithSyncer() — switches the default Batcher to a sync
    span processor. Right for teaching demos and tests; production
    servers should stay on the batched default and handle SIGTERM with
    explicit ForceFlush + Shutdown.

NewTracerProvider(nil) panics — silently constructing a
TracerProvider that emits no spans loses signals without surfacing
the misconfig. Matches NewProvider(nil)’s fail-fast.

Future options (WithDeploymentEnvironment, WithServiceVersion,
WithResource(*resource.Resource) escape hatch) compose against the
same internal config — file when a consumer asks. The
examples/common/otel/ sub-package already migrated to consume this
helper; future P6 surface examples (issues 658 / 659 / 660 / 664)
inherit the cleanup without any extra plumbing.

The pattern is identical across all five: each ext/ surface optionally
accepts a core.TracerProvider and instruments its own work, depending on
the core abstraction only — never ext/otel. Same composition shape
ext/events uses for core.Claims. Noop / nil = zero overhead.

examples/common/otel.SetupTelemetry — landed (issue 666; PR 684 + PR 689)

The example layer now ships a uniform, env-gated observability
surface so every example presents the same --exporter / --otlp-endpoint
flag pair regardless of which SEP it demos. examples/common/otel/setup.go:

tp, shutdown, err := commonotel.SetupTelemetry(ctx,
    commonotel.WithExporter(*tel.Exporter),
    commonotel.WithOTLPEndpoint(*tel.OTLPEndpoint),
    commonotel.WithServiceName("my-example"),
)
defer shutdown(context.Background())
common.RunServer(common.ServerConfig{TracerProvider: tp, ...})

The EXPORTER selector is four-valued:

Value Behavior
"" (default) core.NoopTracerProvider{} + no-op shutdown. Zero overhead.
"stdout" stdouttrace exporter (sync). Demo / teaching mode.
"otlp" TCP-probe the endpoint; on success, otlptracegrpc exporter. On failure: Noop with warning log.
"auto" TCP-probe the endpoint; on success, otlptracegrpc exporter. On failure: Noop silently (operator opted into maybe-on-maybe-off semantics).

Three load-bearing details:

  • TCP probe gates OTLP. otlptracegrpc.New is lazy and returns a
    non-nil exporter even when the endpoint refuses; without the
    500ms TCP probe in probeOTLPEndpoint, the “dial-failure → Noop”
    contract wouldn’t fire (failures would surface later on first
    Export, way past just demo startup).
  • Service-name convention. Server side uses <example-name>;
    walkthrough (client) side uses <example-name>-host. Grafana’s
    service filter distinguishes the two halves of each stitched trace.
  • examples/otel/stdout/ is the documented carve-out. Its
    defaultExporter = "stdout" because the demo’s whole purpose is
    showing spans. Every other example defaults to "".

Client-side counterpart:

tel := common.ExporterFromArgs()  // os.Args scan, mirrors common.ServerURL pattern
tp, shutdown, err := commonotel.SetupClientTelemetry(ctx,
    commonotel.WithExporter(*tel.Exporter),
    commonotel.WithOTLPEndpoint(*tel.OTLPEndpoint),
    commonotel.WithServiceName("my-example-host"),
)
defer shutdown(context.Background())
c := client.NewClient(url, info, client.WithTracerProvider(tp))

SetupClientTelemetry pre-sets
WithInstrumentationName("github.com/panyam/mcpkit/client") so spans
group correctly by library in OTel backends.

PR 684 adopted across 10 example servers (auth, elicitation,
file-inputs, fine-grained-auth, list-ttl, mrtr, skills, stateless,
tasks, tasks-v2); PR 689 followed with the same uniform wiring across
9 walkthroughs. Pattern documented in examples/CONVENTIONS.md
§Telemetry wiring.

ext/auth JWT validator instrumentation — landed (issue 658; PR 694)

First P6 surface child to land on the spine. JWTConfig.TracerProvider core.TracerProvider opts the validator into instrumentation:

  • auth.jwks_lookup sub-span wraps each JWKSKeyStore.GetKeyByKid call (with mcp.auth.jwks.kid attribute; records error on lookup failure). Surfaces both cache-hit and cache-miss latency.
  • mcp.auth.* attributes on the active dispatch span (via core.SpanFromContext): mcp.auth.method = "jwt" stamped early (failure paths included), mcp.auth.subject / issuer / scopes / cache_hit set after claims extraction.

ext/auth depends on core.TracerProvider only for its own spans — no compile-time dependency on ext/otel. Nil and core.NoopTracerProvider{} both produce zero spans, zero attributes, zero allocation. Documented in ext/auth/docs/DESIGN.md § Tracing.

oneauth v0.1.17 wiring — landed (PR 699)

Threads oneauth’s own internal spans through ext/auth so an end-to-end auth trace shows the inside of the JWKS call too. Three layered helpers enable this without abstraction leakage:

  • ext/otel.Provider.OTelTracerProvider() trace.TracerProvider — Provider stashes the underlying OTel TP and exposes it. Use when a downstream library needs the OTel SDK type directly.
  • examples/common/otel.UnderlyingOTelTP(tp core.TracerProvider) trace.TracerProvider — type-asserts to *mcpotel.Provider; returns nil for Noop or non-mcpotel providers (oneauth’s options no-op cleanly on nil).
  • ext/auth.JWTConfig.OneauthTracerProvider trace.TracerProvider — when set, threaded via keys.WithTracerProvider on the JWKSKeyStore so oneauth’s internal HTTP / parsing / signature-verify work emits spans on the same OTel pipeline.

End-to-end trace shape (when both TracerProvider AND OneauthTracerProvider are wired):

client.tools/call          (mcpkit/client)
└─ server.tools/call       (mcpkit/server, parent via _meta.traceparent)
   ├─ auth.jwks_lookup     (ext/auth)
   │  └─ oneauth.jwks.refresh
   │     └─ oneauth.jwks.key_lookup
   └─ tool handler work

Tracked in panyam/oneauth#254 (closed in oneauth v0.1.14; mcpkit ext/auth on v0.1.17 after a CI fix that swept all 8 oneauth-using modules in lock-step). Adopter pattern shipped in examples/auth/common/setup.go via WithMCPTracerProvider / WithOneauthTracerProvider variadic options on Env.NewValidator. Demo wires both in examples/auth/main.go.

Apps Bridge trace context relay — landed (issue 660; PR 702)

Bidirectional W3C trace context propagation across the iframe ↔ host postMessage boundary so a browser-side trace (browser OTel SDK / RUM / hard-coded demo traceparent) stitches with the backend tool-call span. Off by default on both sides; adopters opt in independently per side.

TS bridge (ext/ui/assets/mcp-app-bridge.ts):

MCPApp.setTraceContextProvider(() => ({
  traceparent: "00-...-...-01",
  tracestate: "vendor=val",  // optional
}));

The provider is called before every outbound request() / notify(). Result merges into params._meta.traceparent / _meta.tracestate. Caller-set _meta wins (provider is a fallback). Provider throws are caught and logged; the request still sends without propagation. dep-free — no OTel JS dependency.

Go AppHost (ext/ui/app_host.go):

host := ui.NewAppHost(c, bridge, ui.WithTracerProvider(tp))

handleAppRequest extracts inbound params._meta.traceparent via core.ExtractTraceContextFromParams, wraps the forward to the MCP server in an apps.host.forward span parented by the iframe’s traceparent. The outbound client.Call preserves the iframe’s _meta.traceparent on the wire (SEP-414 P3’s caller-set-wins contract) so the server’s P2 dispatch span stitches as a child.

End-to-end trace shape:

iframe traceparent          (browser-side OTel; demo: hard-coded random)
└─ apps.host.forward        (ext/ui AppHost)
   └─ server tools/call     (mcpkit server, SEP-414 P2)
      └─ tool handler work

Demo wiring in examples/apps/vanilla/dice.html (per-page-load random traceparent + window.__MCPAPP_DEMO_TRACEPARENT__ for on-page TraceID copy). Host wiring in examples/host/01-apphost/main.go. Design doc: docs/APPS_DESIGN.md § Tracing across the Apps Bridge.

Open spec question: the Apps Bridge is a non-MCP transport; whether the relay belongs in the Apps spec for cross-SDK interop is open. mcpkit ships the relay; upstream note filed only if working-group interest surfaces.

Events bus trace context relay — landed (issue 683; PR 712 + PR 714)

W3C Trace Context propagates across every gate in the events lifecycle so a yield on replica A and a downstream webhook delivery (or a poll-side replay on replica B) appear in Tempo as one stitched trace. Sister to the Apps Bridge relay — same “non-MCP propagation hop” pattern across a different boundary topology.

Four gates, three carriers, one consistent rule. Each gate picks the carrier appropriate to its boundary:

Gate Carrier Where
1. yield(ctx, data) event.Meta.traceparent (persistent) + ctx (in-process) YieldingSource.yield stamps Meta from ctx; emit hook receives the same ctx
2. emit hook → Emitter.Emit(ctx, event) ctx Register passes the hook’s ctx straight to the configured Emitter
3. WebhookRegistry.Deliver(ctx, event) → outbound HTTP HTTP traceparent header deliver() extracts from ctx (preferred) or event.Meta (fallback for replayed events), stamps the header before client.Do
4. HTTPSource.serveInject (receiving replica) inbound HTTP header → ctx → event.Meta Handler reads the traceparent header, builds core.TraceContext, attaches via core.WithTraceContext, calls s.yield(ctx, data) — closes the round-trip

Caller-preserves rule. If SetMetaFunc pre-stamps event.Meta.traceparent, the yield-time auto-injection is skipped — uniform with core.InjectTraceContextIntoParams and the TS-side Apps Bridge relay.

events.webhook.deliver span (PR 714). WebhookRegistry.WithWebhookTracerProvider(tp) opts the registry into emitting a span around each retry loop. Attributes: webhook.target.id, webhook.url, mcp.event.name, http.method, http.response.status_code, webhook.retry.attempts. RecordError fires on retries-exhausted with the categorical bucket. Nil = Noop, zero overhead. Live-verified against the LGTM stack: span landed with all attributes set, 4 attempts counted, STATUS_CODE_ERROR on the failure path, duration matched the 0.5s + 1s + 2s backoff schedule exactly.

Server.Broadcast(ctx, ...) (PR 714 signature; PR 722 body). PR 714 widened the signature; PR 722 (issue 715) made the body actually consume ctx — when ctx carries a non-zero core.TraceContext, the inbound traceparent / tracestate are injected into notification params under _meta (via core.InjectTraceContextIntoParams) before fan-out. Existing caller-set _meta.traceparent wins. Per-transport broadcast callbacks gained ctx as a forward-compat seam — unused inside today because the per-request trace_middleware.go wrap (core.WrapSessionNotifyFunc) targets sc.notify (the handler-facing notify on a per-request SessionCtx), not the transport-level base notifyFunc dispatchers expose to broadcasts. Injecting once at the Server level keeps tracing concerns out of the transport-level loops while still stitching SSE-pushed notifications into the originating trace.

End-to-end trace shape (multi-replica with webhook fanout):

yield(ctx, data) on replica A
└─ event.Meta.traceparent stamped
   └─ emit hook (ctx) → LocalEmitter → WebhookRegistry.Deliver(ctx, event)
      └─ events.webhook.deliver span (PR 714)
         └─ HTTP POST to replica B with traceparent header
            └─ replica B HTTPSource.serveInject reads header
               └─ yield(ctx, data) on replica B (same TraceID)

Demo wiring lives in examples/whole-enchilada/events/event-server/main.go (events.WithWebhookTracerProvider(tp) threaded against the existing commonotel.SetupTelemetry TP). Adopter sweep (PR 714) updated examples/events/discord and examples/events/telegram to the new yield(ctx, ...) / SetMetaFunc(ctx, ...) signatures. Cross-replica peer-fanout Emitter (Redis pubsub) tracked on panyam/mcpkit#634 + panyam/mcpkit#639 with concrete design comments added.

Events fanout span emission — landed (issue 724; PR 730)

Companion to the bus-relay work above. While 683/712/714 handled propagation (trace context flowing across the lifecycle gates), 724 handles emission on the per-yield fanout itself. Adopters opt in via events.Config.TracerProvider core.TracerProvider — Register threads it onto every Source implementing the new TracerProviderInstaller interface (YieldingSource does; TypedSource authors can implement it identically).

events.Register(events.Config{
    Server:         srv,
    Sources:        []events.EventSource{chatSrc, alertSrc},
    Webhooks:       webhooks,
    TracerProvider: tp, // ← new in PR 730
})

Span shape: one events.fanout span per emitted event (NOT per subscriber), parented by the yield ctx. Attributes:

Attribute Meaning
mcp.event.name the EventDef.Name
mcp.event.id the assigned EventID (cross-referencing)
events.subscribers.total snapshot subscriber count at fanout time
events.subscribers.delivered count that passed Match
events.subscribers.dropped_by_match count where Match returned false
events.transforms.applied count where Transform actually modified the event

Design call locked in: one span per yield, NOT per subscriber. Option (a) from the original issue body (events.match / events.transform per Match/Transform invocation) was deliberately NOT shipped — would scale linearly with subs × events and need sampling design. The aggregate counts are the diagnosable shape; operators see “this yield went to 10 subs, 7 dropped by Match” without span-volume risk. If per-subscriber detail ever becomes load-bearing, option (a) can land as a separate opt-in on the same TracerProvider seam.

Two load-bearing optimizations on the hot path:

  • Zero-subscriber guard. len(subs) == 0 skips span emission entirely. Idle sources (feeders with no subscribers registered yet) are dominant — emitting empty fanout spans every feeder tick would flood Tempo with noise.
  • Noop short-circuit. tp.(core.NoopTracerProvider) type-assertion avoids the StartSpan call entirely on the unconfigured path. Per-yield cost on the unconfigured path is one type-assertion.

NOT in scope (deferred to future PRs if anyone asks): webhook-side Match/Transform attributes on the existing events.webhook.deliver span (PR 714); poll-side Match/Transform aggregate attributes on the events/poll dispatch span. Both are covered by existing spans; decorating with more attributes is drive-by.

Test pattern: reuses the fakeTP recording machinery from webhook_span_test.go so the assertion style stays consistent across event-surface span tests. Tests cover attribute shape, drop-by-Match, transforms-applied, zero-subscriber skip, parent stitching, Noop overhead, and the Register install path.

End-to-end trace shape:

tools/call (dispatch, SEP-414 P2)
└─ events.fanout (PR 730 — one span per yield, parented here)
   ├─ events.subscribers.total = 10
   ├─ events.subscribers.delivered = 3
   ├─ events.subscribers.dropped_by_match = 7
   └─ events.transforms.applied = 3
   └─ events.webhook.deliver (PR 714 — per webhook target, sibling span via existing seam)

Doc: experimental/ext/events/README.md § Tracing across the events bus + examples/events/kitchen-sink/README.md § Tracing.

ext/tasks task lifecycle spans — landed (issue 659; PR 719)

Final P6 surface child. A tools/call that creates a task spawns a background goroutine whose work outlives the create span — parent-child doesn’t fit, so the tasks runtime uses span links instead. The dispatch span for the tools/call ends in a few ms; task.execute runs as a NEW root trace (potentially for hours) with a Link back to the create span. Every later tasks/get / tasks/update / tasks/cancel dispatch span gets an AddLink back to the same create span so a backend can pivot from any poll into the whole lifecycle.

Wiring: tasks.Config.TracerProvider core.TracerProvider. Nil or core.NoopTracerProvider{} (the default) skips the install — zero overhead, no spans, no allocation. ext/tasks depends on the core abstraction only; no compile-time dep on ext/otel.

tasks.Register(tasks.Config{
    Server:         srv,
    TracerProvider: mcpotel.NewProvider(otelTP),
})

Span shape:

Span Lifecycle Parent Links Attributes
tools/call (dispatch) request scope inbound _meta.traceparent — SEP-414 P2 dispatch attrs
task.execute goroutine lifetime none (new root via WithNewRootSpan) → tools/call create span mcp.task.id, mcp.task.status (stamped at End)
tasks/get / tasks/update / tasks/cancel request scope inbound _meta.traceparent → tools/call create span (AddLink) SEP-414 P2 dispatch attrs

Status stamping: the goroutine’s defer reads the final status from the store and stamps mcp.task.status (completed / failed / cancelled / input_required) on the span, then Ends. SEP-2663 semantics: handler-returned errors ((nil, err)) map to completed with IsError=true on the inner ToolResult — RecordError on the span is reserved for protocol-level failures (mwErr, resp.Error, unexpected result shape, panic recover).

Sync-as-completed path is deliberately silent. wrapSyncAsCompletedTask (a sync-returning handler under TaskSupport=optional/required) runs inside the create span — emitting a zero-duration task.execute for the wire-uniformity wrapper would be noise. Only the GoAsync path (the work that actually escapes the request span) gets task.execute.

Lifetime of originTC (the runtime map that stashes the create-span TraceContext per taskID): outlives the activeTask entry. Polls landing on a terminal task still get the AddLink back to the create span. Matches the existing pattern for taskErrors / creatorToolForTask.

End-to-end trace shape:

tools/call (dispatch)                         (mcpkit/server, SEP-414 P2)
└─ [link] task.execute  ────────────────►     (root trace, ext/tasks goroutine, mcp.task.id, mcp.task.status)
                                              (tool handler work runs inside task.execute)

tasks/get (dispatch, link → original tools/call create span)
tasks/update (dispatch, link → original tools/call create span)
tasks/cancel (dispatch, link → original tools/call create span)

End-to-end materialization is proven by ext/otel/provider_test.go’s coverage of the WithNewRootSpan scrub + StartSpanLinked + AddLink paths against a real OTel SDK exporter. ext/tasks/trace_test.go uses a recording fake core.TracerProvider + core.LinkedTracerProvider to prove ext/tasks calls the contract correctly without dragging ext/otel into its module graph.

MRTR multi-round trace stitching — landed (issue 682; PR forthcoming)

SEP-2322 MRTR splits one logical operation across N JSON-RPC dispatches (round 1: server returns InputRequiredResult → client gathers input → round 2: client retries with inputResponses). Each round’s tools/call mints its own span on each side, so without intervention an MRTR operation produces N unrelated traces — an operator looking at the final ToolResult’s trace cannot navigate to the input-gathering work that produced it.

This is fundamental once stateless (SEP-2575) replaces stateful as the default wire — stateful’s ctx.Sample / ctx.Elicit are deprecated per SEP-2577 and will go away in v0.4. MRTR becomes the canonical input-gathering pattern; the trace story has to follow.

Wire shape: rounds 2+ carry the round-1 traceparent in a new _meta.io.modelcontextprotocol/tracelink field. The server’s trace middleware reads it and calls core.SpanFromContext(ctx).AddLink(...) on the round-N dispatch span, stitching the operation across separate W3C traces.

Round 1:                              Round 2 (and beyond):
client.Call("tools/call", args)       client.Call("tools/call", args+inputResponses)
  ↓                                     ↓
_meta: {                              _meta: {
  traceparent: T1                       traceparent: T2,
}                                       io.modelcontextprotocol/tracelink: T1,  ← NEW (issue 682)
                                      }
  ↓                                     ↓
server span (TraceID T1)              server span (TraceID T2)
                                        AddLink → T1  ← server stitches the operation

Star semantic — rounds 2, 3, 4… all link back to round 1, NOT the immediately-previous round. Backends show round 1 as the anchor regardless of how deep the MRTR loop goes. One click navigates from any round to the originating request.

Vendor-namespaced field: _meta.io.modelcontextprotocol/tracelink (not bare _meta.tracelink) because W3C doesn’t define a tracelink field — bare names are reserved for W3C-defined keys. mcpkit-specific until upstream WG agrees on a cross-SDK shape.

Client-side identity capture: CallToolWithInputs uses the new core.WithCapturedTraceContext ctx-holder primitive. The client trace middleware writes round-1’s outbound TraceContext into the holder; CallToolWithInputs snapshots it and stamps it onto every subsequent round’s params via core.InjectTraceLinkIntoParams. Tracestate is intentionally NOT part of the link payload — link is identity-only, not full propagation context. Client.callImpl(ctx, ...) (a ctx-aware refactor of Client.Call) is what threads the holder through the middleware chain; Client.Call itself stays ctx-free for back-compat by passing context.Background().

Considered alternatives:

  • Embed traceparent in requestState — works, but requestState was designed as opaque from the client’s perspective. Baking tracing semantics into it muddies the contract.
  • Continue an outer span across rounds on the client side — simpler (no wire change), but the outer span’s duration would include user-paced gather-input time, which can be minutes. Distorts span duration semantics. Rejected.

Spec engagement TBD: this lands as mcpkit-internal first; the field name + semantic could be a candidate for upstream MCP WG standardization for cross-SDK interop. See the PR description for the working-group post draft.

End-to-end correctness proof: ext/otel/mrtr_tracelink_e2e_test.go drives a real CallToolWithInputs against a real OTel-SDK-backed server + client, then asserts the recorded round-2 server span has an OTel Link whose TraceID matches round-1’s.

Skills observability — landed (issue 748; PR forthcoming)

Tool calls have natural telemetry: every tools/call is one wire event → one server-side dispatch span. Skills (SEP-2640) don’t — the host fetches a manifest once via resources/read, caches it client-side, then activates it N times invisibly. Without intervention, post-cache activation is unobservable to either side.

Issue 748 closes the gap in two layers.

Layer 1 — server dispatch span enrichment. server/trace_middleware.go now stamps the resources/read span with mcp.resource.uri (always, for any URI) plus mcp.skill.uri / mcp.skill.path / mcp.skill.file when the URI uses the skill:// scheme. Mirrors the existing tools/call → mcp.tool.name pattern. Server-side dashboards can now chart fetch volume + error rate per skill.

mcp.skill.path and mcp.skill.file only populate for SEP-2640 manifest URIs (terminal /SKILL.md) where the path/file boundary is unambiguous from the URI alone. Non-manifest URIs (e.g. skill://pdf-processing/references/FORMS.md) surface as mcp.skill.uri only — SEP-2640 documents that the boundary in those cases requires external knowledge from the discovery index or a prior manifest read.

Layer 2 — client-side ext/skills.Client instrumentation. NewClient(mcp, opts...) is variadic; new WithTracerProvider(tp) option wraps the read-path methods in spans:

Method Span name Attributes
ListSkills skills.list mcp.skill.uri (index URI), mcp.skill.count on success
ReadSkillURI skills.read mcp.skill.uri
ReadSkillManifest skills.read_manifest mcp.skill.uri, mcp.skill.path, mcp.skill.name
ReadAndVerify skills.read_and_verify mcp.skill.uri, mcp.skill.expected_digest, mcp.skill.digest_verified

Stitching is automatic: the client read span runs inside ctx that already carries _meta.traceparent from the existing W3C trace context propagation (PR 644 / PR 649 / PR 652), so the server-side dispatch span (now enriched per Layer 1) lands as a child of the client wrapping span in the same trace.

The activation hook — purely SDK-side, no wire change. Client.Activate(ctx, uri, opts...) ActivationEvent is the answer to the “skill used” telemetry the wire can’t capture. Hosts call it at the point in the agent loop where the skill enters model context. Side effects:

  • Emits an instant skills.activate span (Start + End back-to-back — activation is point-in-time, not a duration) carrying mcp.skill.uri, mcp.skill.path (when the URI parses as a manifest URI), and mcp.skill.activation.reason when WithReason(...) is supplied.
  • Invokes the optional WithActivationHook(fn) callback synchronously, so non-OTel hosts can feed their own telemetry pipeline (structured logging, internal counters, alternate tracing) without configuring a TracerProvider.
  • Returns the ActivationEvent{URI, Reason, Timestamp} to the caller for pull-style stamping.

Activate accepts any string URI — it never blocks dispatch or validates input. Designed for the agent-loop case where the URI may be a manifest URI, a sub-file URI, or even an opaque identifier the host minted for an in-process bundle.

Considered alternatives:

  • Wire-level notifications/...skills/activated — a cross-process notification carrying {uri, digest, reason, _meta.traceparent} so the server can record activation events from connected clients. Tractable but cross-cuts SEP-2640 spec territory and requires WG engagement before minting the method name. Tracked as issue 749 — a separable opt-in extension on ext/skills that can ride atop Client.Activate without breaking callers when (if) the upstream method name lands.
  • Provider-side activation telemetry — N/A. The provider handles resource reads; activation is a client/host concept.
  • Activation counter on the issue 735 MeterProvider seam — useful follow-up. The TracerProvider half ships in 748; counter wiring can land cleanly in a small follow-up PR.

Coverage:

  • server/trace_middleware_test.go — three tests pin Layer 1: skill manifest URI emits all four attrs; non-skill URI emits mcp.resource.uri only; non-manifest skill URI emits mcp.skill.uri only.
  • ext/skills/client_trace_test.go — eight tests pin Layer 2: each wrapped method emits the right span + attrs; Activate fires hook + span; Activate omits the reason attr when not supplied; hook fires independently of tracer install; zero-overhead path (no WithTracerProvider) emits no observable spans.
  • End-to-end trace context propagation across resources/read is the same code path the MRTR e2e already exercises for tools/call (_meta.traceparent injection in dispatch is method-agnostic). A skills-specific real-SDK e2e is deliberately omitted — it would require either inverting the ext/otel-doesn’t-import-ext/skills layering or adding the real OTel SDK as a test-only dep on ext/skills.

Runnable demo: examples/skills/walkthrough.go exercises the wrap span + Activate path with the --exporter=stdout (or --exporter=otlp) flag pair. Run with EXPORTER=stdout just serve + EXPORTER=stdout just demo.

Downstream consumers of the Phase 1 contract

  • experimental/ext/events/ cross-replica bus (issue #629). The
    EventBus envelope will carry traceparent / tracestate using the
    core.MetaKey* constants, so a trace started on replica A and
    delivered from replica B stitches together once an OTel adapter is
    wired. The seam consumes core.TracerProvider directly — no
    ext/otel import in the base events module, mirroring how events
    already consumes core.Claims without depending on ext/auth.
  • server/ topology preflight (issue #642). The capability
    contract declares whether a configured seam supports trace
    propagation; the declaration uses the core.TracerProvider
    interface to keep the contract dep-free.