The three pillars of observability are metrics, logs, and traces. This taxonomy has been broadly accepted for about a decade, popularized by the distributed systems community and now enshrined in products like OpenTelemetry, Datadog, and Honeycomb. It is a useful model. It is also a model designed for synchronous, request/response systems, and it has a structural gap when applied to event-driven architectures.
That gap is schema integrity. Not as a niche concern for data teams, but as a first-class observability primitive for any system where services communicate asynchronously through durable event logs.
What the Three Pillars Actually Measure
To understand the gap, it helps to be precise about what each pillar covers.
Metrics capture aggregate system behavior over time: throughput, latency, error rates, resource utilization. In a Kafka deployment, your metrics tell you consumer lag per topic and partition, broker disk utilization, produce and consume rates, and replication lag. These are infrastructure-level signals.
Logs capture discrete events emitted by application code: exceptions, state transitions, structured audit records. A Kafka consumer that encounters a deserialization error will log it. A consumer that encounters a semantically wrong event and silently produces a bad output will not log anything, because from its perspective nothing went wrong.
Traces capture request paths across service boundaries: spans, latency breakdowns, dependency chains. In an event-driven system, traces get complicated quickly because the causal chain between producer and consumer is asynchronous and the propagation context (trace IDs in event headers) requires active instrumentation that many teams do not maintain end-to-end.
None of these three pillars asks the question: does the data inside these events look like it should?
What Schema Integrity Monitoring Actually Is
Schema integrity monitoring, as we think about it, is not just schema validation. Schema validation is binary: either an event conforms to a registered schema or it does not. Most schema registries enforce this at produce time for registered topics.
Schema integrity monitoring is the ongoing observation of event content against a behavioral baseline. It asks: is the distribution of values in this field consistent with what we have seen historically? Is a field that was never null suddenly null for 40% of events? Has a field that reliably contained values in the range 0-100 started producing values in the range 0-1 (suggesting a unit conversion bug)? Did a timestamp field shift from seconds to milliseconds epoch, which produces structurally valid integers that are semantically two orders of magnitude off?
These are not schema violations in the Avro or Protobuf sense. They are behavioral deviations that break downstream logic in ways that look, from the infrastructure layer, like everything is fine.
Why Event-Driven Systems Are Specifically Vulnerable
In a request/response system, a broken API response usually causes an immediate failure. The calling service gets a 500, or a deserialization error, or a null pointer exception that surfaces in logs within seconds. The feedback loop is tight. Teams learn about breakage fast.
In an event-driven system, the producer and consumer are decoupled in time. A producer emits events that sit in a Kafka partition for potentially hours before a consumer processes them. The consumer may be a batch analytics job that runs once daily. Or the consumer may process events in real time but produce a downstream write that only surfaces as wrong data when a human looks at a report the next morning.
This temporal decoupling means the feedback loop for data quality issues is measured in hours, not seconds. By the time someone notices the anomaly, the bad events have propagated through multiple downstream systems. Backfilling is expensive. In some cases, like financial transaction records, it is not straightforwardly possible.
We are not saying event-driven architectures are harder to operate than synchronous ones overall. In many dimensions they are more resilient: a producer can keep emitting even when consumers are down, and consumers can catch up from durable storage. The tradeoff is that the temporal decoupling that creates this resilience also removes the natural error feedback that synchronous systems provide for data quality issues.
The Gap in Practice: A Scenario
Consider a payment processing pipeline running on Kafka. A producer service emits transaction events to a topic. An enrichment consumer reads those events, appends currency conversion rates, and writes to a downstream settlement topic. A settlement processor reads from the downstream topic and books transactions in the ledger.
A developer working on the producer service changes the amount field from a decimal representation (e.g., 142.50) to a minor-unit integer (e.g., 14250) to align with a new internal standard. This is a known pattern in payment systems. The developer updates the producer. The enrichment consumer was not notified, so it reads 14250 and applies a currency conversion to it. The resulting settlement amounts are now 100x the correct values.
From the metrics layer: consumer lag is normal, produce rate is normal, no broker errors. From the logs layer: no exceptions, the enrichment consumer parsed every event successfully. From the traces layer: all spans closed cleanly. From schema validation: the field is still an integer, still present, still within what the Avro schema allows. Everything looks healthy until someone notices that transaction amounts in the ledger are wrong.
Schema integrity monitoring would have flagged this within minutes: the distribution of values in amount shifted by two orders of magnitude. The baseline model had seen weeks of values in the range 1-50,000 (cents). Suddenly it is seeing values in the range 100-5,000,000. That is an anomaly, and the right response is to quarantine those events and alert before they hit the settlement topic.
How This Fits into an Observability Stack
We are not positioning schema integrity monitoring as a replacement for metrics, logs, and traces. Those three pillars are essential and well-solved by existing tooling. What we are saying is that in event-driven architectures, there is a fourth signal that the existing three do not cover: the behavioral health of event content.
Practically, this means adding a monitoring layer that sits between your event broker and your consumers. For Kafka-based stacks, this looks like a read-side observer that consumes from the same topics as your application consumers but does not process events, only analyzes them. It maintains field-level baselines per topic, computes deviations as new events arrive, and emits anomaly signals that can feed into your existing alerting infrastructure via PagerDuty, Slack, or your incident management tooling of choice.
The integration point with your existing observability stack is through those alert signals. A schema integrity alert shows up alongside your infrastructure alerts. The difference is that it contains information the infrastructure layer cannot provide: which field changed, how it changed, and what the before/after distribution looks like. That context collapses the time-to-diagnose for a class of incidents that would otherwise take hours to trace.
What Teams Actually Do Today
Most teams handling this problem today rely on a combination of contract testing in CI and ad hoc monitoring scripts. Contract testing is valuable and we encourage it. The problem is that contract tests run against committed schema definitions, not against the actual behavior of a live producer under production traffic. A producer can pass all its contract tests and still emit unexpected null values because of a configuration flag that is only set in production.
Monitoring scripts are usually written after an incident, to watch for the specific pattern that just caused pain. They accumulate as a collection of one-off checks that are not generalizable and create their own maintenance overhead.
Neither approach provides real-time behavioral observation of live traffic. That is the gap. The good news is that it is a tractable gap: the data is all there in the stream, it just needs something watching it that knows what normal looks like.