This is a post about how we actually built the core of Streamforge's schema monitoring. Not the concept, but the implementation decisions: why we built a probabilistic baseline from live traffic rather than relying on registered schemas, how we handle the sampling tradeoff, and what it took to make the baseline adaptive without making it so adaptive that it just learns drift as normal.
The starting question was: given a Kafka topic with a registered Avro schema, what does a full picture of "schema health" look like beyond "does the message conform to the Avro envelope"? The registered schema tells you the types. It doesn't tell you the expected nullability rates, the typical value ranges, the field presence rates for optional fields, or the statistical distribution of enum values. Those are the things that drift when a producer changes behavior without changing the schema version.
Why a Registered Schema Isn't Enough for Baseline
An Avro schema might define a field as union[null, string], meaning it can be null or a string. That's structurally valid. But on a specific topic, this field might be null 0.2% of the time under normal conditions. If it starts being null 30% of the time, the Avro schema doesn't flag it. Consumer behavior changes without any schema version bump.
To detect that kind of drift, you need a baseline built from observed behavior, not from the schema definition. We call this the schema fingerprint: a statistical representation of what "healthy" looks like for each field on a topic, derived from sampling live traffic over a learning window.
The fingerprint isn't a replacement for the registered schema. It's a layer on top of it. Schema Registry catches structural violations at produce time. The fingerprint catches semantic drift at runtime.
Sampling Strategy
Sampling at high throughput is a real engineering challenge. A topic producing 50,000 events per second can't be inspected in full without significant infrastructure overhead. We needed a sampling strategy that gave statistically reliable fingerprint estimates without consuming proportionally more resources as throughput grows.
We settled on a reservoir sampling approach with a per-topic sample rate that scales as O(log N) with topic throughput, not O(N). For a high-throughput topic, we sample roughly 0.1-0.5% of events into the fingerprint computation window. For a low-throughput topic (under 100 events per minute), we sample all events because the volume is small enough to inspect without filtering.
The challenge with low sampling rates is that rare field patterns can be missed. If a field is populated in only 0.05% of events normally, and our sampling rate is 0.1%, we might see that field in roughly half the samples. This creates noise in the nullability estimates for rare optional fields. We handle this by widening the confidence bands for fields with low historical presence rates: the anomaly threshold for a field that's typically present in 0.1% of events is much looser than for a field that's typically present in 99% of events.
What the Fingerprint Captures Per Field
For each field in the observed schema, the fingerprint maintains:
- Presence rate: what fraction of events contain this field. For required fields this should be close to 1.0; for optional fields it captures the observed population rate.
- Type distribution: for union types, what fraction of events carry each type variant. This catches the
string vs floatdrift described above. - Nullability percentile: the p50, p95, and p99 of the null fraction in rolling windows. Drift in nullability is often gradual, so tracking the full percentile distribution matters.
- Value range: for numeric fields, the p5 and p95 of observed values. Not min/max, because extreme outliers skew those. The interquartile range stays stable under legitimate traffic variance while shifting when there's a data issue.
- Cardinality estimate: for string fields that look like enum-style categoricals, the approximate number of distinct values and the distribution over those values. A field with 5 known values suddenly producing a 6th value at 40% rate is a fingerprint anomaly.
We use HyperLogLog for cardinality estimates because exact cardinality is expensive to track for high-cardinality string fields, and we don't need it. We need to know when a previously-closed vocabulary opens up, which HyperLogLog detects reliably with much lower memory footprint than an exact counter.
The Learning Window and Update Cadence
The fingerprint is built from a learning window of observed traffic. The window length is configurable per topic, with a default of 7 days. That window is long enough to capture weekly traffic patterns (business-hours vs overnight, weekday vs weekend) without being so long that it's slow to adapt to genuine schema evolution.
After the initial 7-day learning window, the fingerprint updates on a rolling basis. New observations are incorporated with a decay-weighted update that gives more weight to recent observations. The decay factor is tunable: a fast decay (aggressive learning) adapts quickly to schema changes but also learns from drift. A slow decay (conservative learning) is more stable but takes longer to recognize intentional evolution.
This is the fundamental tension in any adaptive baseline system: you want it to learn from legitimate evolution but not from anomalous drift. Our approach to managing this tension is to separate the update path for confirmed-normal observations from the update path for anomalous observations. When Streamforge detects an anomaly and routes events to quarantine, those events don't contribute to the fingerprint update. Only events that pass the anomaly check update the baseline. This prevents the baseline from learning drift as normal during an active incident.
Handling Multi-Modal Traffic Patterns
Some topics have bimodal or multimodal field distributions that don't fit a simple single-distribution fingerprint. An orders topic might carry both domestic and international orders, where the shipping_region field has very different value distributions for each. If international orders make up 20% of traffic and domestic 80%, a single presence rate for customs_declaration_id (present on international orders, absent on domestic) will show 20% presence rate normally. An anomaly that shifts it to 5% might be a data issue with international orders, but the single-distribution baseline doesn't capture that.
We handle this with conditional fingerprinting: for topics where we detect that field distributions are correlated with another field value (e.g., order_type == "international"), we maintain separate fingerprint tracks conditioned on that discriminator field. This requires automatically detecting the correlation, which we do via a mutual information computation over the sampled field values during the learning window.
Conditional fingerprinting adds complexity and is not enabled by default. It activates automatically when the mutual information between a proposed discriminator field and other fields exceeds a threshold during the learning window. We've found it necessary for topics that carry mixed event subtypes without a formal union schema.
False Positive Calibration
The hardest problem in building the fingerprint system was false positive calibration. An anomaly threshold that's too tight generates noise constantly. An anomaly threshold that's too loose misses real drift. The right threshold depends on the statistical properties of the baseline, which vary by field and by topic.
We calibrate anomaly thresholds using a holdout validation approach during the learning window. The learning window is split: the first 5 days build the initial fingerprint, and the last 2 days serve as a validation set. We compute what the anomaly detection would have fired on during the validation period with different threshold settings and choose the threshold that produces fewer than N false positives per week for that topic, where N is configurable.
This is not perfect. It's subject to the quality of the learning window: if the learning window contains genuine anomalies that were never flagged, those anomalies get baked into the baseline as normal. This is why we also expose the raw fingerprint statistics to users, so they can review and prune the baseline if they know the learning window contained a known incident.
Where the Model Falls Short
The probabilistic fingerprint works well for field-level drift within a single topic. It doesn't work well for cross-topic correlation failures: if two topics are supposed to share a key format and one of them drifts, neither topic shows a within-topic anomaly. The anomaly only appears when you try to join them and the keys don't match.
Cross-topic correlation monitoring is on our roadmap. The technical approach is to define join assertions between topics and evaluate them in a sliding window: "for every event on topic A with key K, there should be a corresponding event on topic B with the same key within W minutes." When that assertion fails at a rate above the baseline miss rate, it's a correlation failure signal. We have the design worked out. The sampling complexity of doing this at scale for many topic pairs is the implementation challenge we're working through.