The incident had been open for fifty-three minutes when someone finally said what everyone was thinking: We have all the data; we just can't connect it. The payment service was throwing intermittent errors — not enough to trip the availability SLO, but enough that a percentage of users were seeing failed transactions. Metrics showed a latency spike on the order service starting about eight minutes before the errors appeared. Logs from the payment service showed connection timeouts. Traces showed… nothing useful, because the team responsible for the order service had deployed a new version two weeks earlier and quietly dropped the trace propagation header in the process.Three separate systems, three separate stories, no shared thread to pull. The investigation turned into a meeting where people read log lines aloud to each other across a screen share, manually comparing timestamps and trying to reconstruct a sequence of events that a properly correlated observability stack would have surfaced in thirty seconds. They found the root cause eventually. It took seventy-eight minutes and four engineers.That experience is not unusual. It's practically the default outcome when observability grows organically, which it almost always does. And the solution is less about which tools you choose and more about one specific architectural decision that most teams get wrong or skip entirely.The Root Problem: Three Data Models With No Shared KeyLogs, metrics, and traces were built by different communities solving different problems, and they reflect that history in their data models. Metrics are aggregates, they deliberately discard individual event identity in exchange for efficient storage and fast queries. Logs are individual events tied to a process, timestamped and structured to varying degrees depending on who wrote the logging code. Traces are causally linked spans representing work that crosses service and process boundaries, identified by a trace ID that's meaningless unless every service in the call chain propagates it correctly.The fundamental problem is that none of these three systems has a native join key to the other two. Timestamp is the obvious candidate, and it's also deeply unreliable at the granularity where it matters when you're trying to correlate a specific request's log lines with the metric anomaly that followed and the trace that explains why. Millisecond clock skew across distributed services makes timestamp joins fragile enough to mislead more than they help.What you actually need is a correlation ID that's first-class in all three signal types simultaneously. In practice, that means the trace ID generated at the request boundary and propagated through every downstream call needs to also appear in log lines and be linkable from metric exemplars. This sounds straightforward. The implementation is where things fall apart.Why Trace ID Propagation Breaks in PracticeThe failure mode I see most often isn't that teams don't know about trace propagation. It's that they implement it inconsistently across a fleet of services that were instrumented at different times, by different people, using different libraries. Service A uses the W3C traceparent header. Service B was instrumented two years ago and uses a custom X-Request-ID header that predates OpenTelemetry. Service C is a third-party dependency that doesn't propagate anything. Service D does propagate the trace ID but doesn't include it in its structured logs because the developer who added logging didn't know about the tracing setup.The result is a trace that looks complete in the tracing UI but is actually missing three hops, combined with logs that have no trace ID field and metrics with no exemplars. You can see each signal in isolation. You cannot move between them programmatically.The fix requires treating trace propagation as an infrastructure concern rather than an application concern. Concretely, for HTTP services, this means running a propagation middleware or sidecar that reads the incoming traceparent header, generates one if absent, and injects it into both the outgoing request context and the structured log fields before any application code runs. In a Kubernetes environment, a service mesh can handle the propagation layer, but you still need to ensure the trace ID reaches the application's logging context.import loggingfrom opentelemetry import tracefrom fastapi import Requestfrom starlette.middleware.base import BaseHTTPMiddlewarelogger = logging.getLogger(__name__)class TraceContextMiddleware(BaseHTTPMiddleware): async def dispatch(self, request: Request, call_next): span = trace.get_current_span() sc = span.get_span_context() trace_id = format(sc.trace_id, "032x") if sc.is_valid else "N/A" span_id = format(sc.span_id, "016x") if sc.is_valid else "N/A" # Inject trace/span IDs into structured log context log = logging.LoggerAdapter(logger, extra={ "trace_id": trace_id, "span_id": span_id, "service": "your-service-name", }) # Attach logger to request state for use in handlers request.state.log = log response = await call_next(request) return response#Registering it is FastAPI:from fastapi import FastAPIapp = FastAPI()app.add_middleware(TraceContextMiddleware)@app.get("/checkout")async def checkout(request: Request): log = request.state.log log.info("Processing checkout request") # trace_id and span_id are automatically includedThis pattern ensures that every log line produced during a request automatically carries the trace ID, without requiring individual developers to remember to include it. The middleware does the work once, and the entire service benefits. The equivalent pattern exists in most language ecosystems; the implementation details vary, but the principle doesn't.Metric Exemplars: The Missing Link to TracesGetting trace IDs into logs solves half the correlation problem. The other half is connecting metrics to traces, which is where most observability setups still have a gap. A metric tells you that p99 latency on the checkout service spiked at 14:32. It doesn't tell you which specific requests were slow or what their trace IDs are so you can examine the traces directly.Prometheus introduced exemplars to solve exactly this problem. An exemplar is a sample data point attached to a metric observation that carries additional labels, specifically, a trace ID. When you record a latency observation, you also attach the trace ID for that request. The result is that you can look at a histogram showing a latency spike, click on the spike, and jump directly to a representative trace from that time window without any manual searching.# Recording a histogram observation with an exemplar (Python)from prometheus_client import Histogramfrom opentelemetry import traceREQUEST_LATENCY = Histogram( 'http_request_duration_seconds', 'Request latency', ['service', 'method', 'status'])def record_request(duration, method, status): span = trace.get_current_span() sc = span.get_span_context() exemplar = {'trace_id': format(sc.trace_id, '032x')} REQUEST_LATENCY.labels( service='checkout', method=method, status=status ).observe(duration, exemplar=exemplar)The practical catch with exemplars is that they require OpenMetrics format support in both the Prometheus scrape configuration and the querying frontend. Grafana supports them, but you need to enable the OpenMetrics scrape format explicitly and use a Grafana version recent enough to render exemplar markers on histogram panels. Teams that skip this setup get metric data without the trace linkage, which means the correlation has to be done manually, which most people won't do under incident pressure.The Clock Skew Problem at ScaleHere's a failure mode that only surfaces at scale: when you're correlating across dozens of services running on hundreds of nodes, clock skew becomes a genuine source of incorrect conclusions. NTP keeps most system clocks within a few milliseconds of each other under normal conditions. Under load nodes, CPU-starved, network-delayed skew can creep to tens of milliseconds or more. When you're trying to correlate a log event with a metric data point from a 15-second scrape window, a 50ms skew is irrelevant. When you're trying to reconstruct the precise ordering of events across six services during an incident that lasted 90 seconds, it can cause you to misread the causal sequence entirely.The mitigation is twofold. First, use the trace ID as the primary correlation mechanism whenever possible, rather than timestamp; trace causality is preserved by the instrumentation itself and doesn't depend on clock accuracy. Second, where you do need to correlate by time across services, apply a correlation window rather than an exact timestamp match, and be explicit in runbooks that timestamp-based correlation carries uncertainty. Teams that treat cross-service timestamps as precise tend to chase phantom causes during incidents.What We'd Do DifferentlyIn hindsight, the single highest-leverage change is mandating trace ID in structured logs from day one as a non-negotiable logging standard, enforced in the shared logging library that all services import. When a trace ID is optional or left to individual developers, it ends up absent in exactly the services where you most need it. The ones that were written quickly, or by contractors, or before the observability standards were written.The second thing worth doing earlier is building a correlation test into the CI pipeline. Not a full end-to-end observability test, just a check that verifies a representative request produces a log line containing a trace ID field that matches the active span. Catching missing trace propagation in CI costs almost nothing. Discovering it during an incident is expensive in exactly the wrong way.When should you not invest heavily in this? If you're running a small system where a single engineer can hold the entire architecture in their head and incidents are rare and simple, the overhead of full three-signal correlation is probably not worth it. A well-structured logging setup with request IDs and clear service attribution will cover most debugging needs. The correlation infrastructure earns its cost when you have multiple teams, services that weren't written by the person debugging them, and incidents that cross more than two service boundaries.Key TakeawaysTrace ID is the only reliable join key across logs, metrics, and traces. Make it first-class in all three signal types, injected automatically at the middleware layer rather than left to individual developers.Metric exemplars connect histograms to specific traces. Enable OpenMetrics format in Prometheus and configure Grafana to render exemplar markers; this is the difference between "latency spiked" and "here's a trace from the spike."Don't rely on timestamps as a correlation mechanism across services. Clock skew at scale makes timestamp joins unreliable for precise causal reconstruction. Use trace causality first, and timestamps as a fallback with an explicit uncertainty window.Enforce trace propagation as infrastructure, not application convention. Middleware, service meshes, and shared logging libraries beat documentation and goodwill every time.ConclusionThe irony of observability at scale is that more data doesn't automatically produce more understanding. Most systems generating incidents are already instrumented. The problem is that the instruments don't speak to each other; they're three separate monologues where you need a conversation.Getting logs, metrics, and traces to correlate reliably isn't primarily a tooling problem. It's a discipline problem: consistent naming, mandatory trace propagation, exemplars wired up correctly, and clock skew accounted for. None of it is technically hard. All of it requires treating observability as a first-class engineering concern rather than something you bolt on after the fact.The question worth sitting with is this: as AI-assisted root cause analysis tools start appearing in observability platforms, the ones that will work best are the ones ingesting clean, correlated signal. If your three data types can't be joined programmatically today, an AI layer on top won't fix that; it'll just be confused faster. How much of your current observability investment is producing signal that's actually queryable across dimensions, and how much is producing data that only makes sense to the person who wrote the service?\\\