Observe what happened.
Three sink families: audit (security-sensitive, durable, backpressure-Wait), operational (durable, backpressure-Wait, the workhorse channel), debug (best-effort, DropOldest under load). Source-transactional observability relay materializes records durably. OpenTelemetry ActivitySource and Meter are first-class. Observability is a peer plane to Explainability — they're never confused.
Different durability tiers. Different semantics.
Audit trails (VadylAuditLog)
Security-sensitive operations. Identity events, capability grant changes, federation contracts, surface installs, schema transitions — all flow here. Backpressure-Wait — never dropped.
Operational trails (OperationalTrail)
Domain mutations and lifecycle events. Backpressure-Wait. The workhorse channel — every canonical lifecycle emitter writes here.
Debug traces (DebugTrace)
Best-effort observability. DropOldest under load — when the platform is hot, debug yields. Use for diagnostic detail, never for compliance trails.
Durable. Coherent. One materializer.
_vadyl_observability_relay
The source-transactional buffer. Records land alongside the source mutation transaction; the relay drain materializes them into terminal sinks. No record is committed without its trail being on the way.
ObservabilityRelayDrainHostedService
Sole materializer of VadylAuditLog / OperationalTrail / DebugTrace. One canonical drain, one canonical pipeline — no second authority materializes observability records.
OpenTelemetry
First-class ActivitySource and Meter. Spans across HTTP, dispatch, provider execution, transactions. Ten-plus canonical metrics — instrument names stable across releases.
VadylMetrics canonical metrics
Read latency, write latency, lifecycle emissions, cache entries, queue depth, trigger duration — all surface as a Meter. Plug into Prometheus, Datadog, anything OpenTelemetry-compatible.
Structured ApiError responses
Every API error is a typed ApiError shape with correlation ID, propagated through ErrorResponseWriter. Operators and clients see the same correlation across HTTP, logs, and trails.
Lifecycle emitters
Every canonical mutation domain has a LifecycleEmitter routing through the canonical platform-event bus + observability sinks. Identity, cache binding, webhook delivery, branching state, billing — all coherent.
Two peer planes. Never confused.
Observability records what happened. Explainability projects why decisions were made. The boundary is structural — no Explainability type references Vadyl.Observability. Reasons are projected directly from canonical authorities, never reconstructed from log lines. Anti-pattern #77 is absolute.
Audit · Operational · Debug
Trail commits with the mutation
Stable instrument names
Anti-pattern #77 absolute
Plug into the OpenTelemetry stack you already run.
ActivitySource for spans. Meter for metrics. Three durability tiers for trails. The same observability everyone gets from canonical Vadyl — your handlers, mutations, decisions all flow through it automatically.