It was 3:14 a.m. when my phone started vibrating across the nightstand. Not an alert from PagerDuty. A Slack message from a product manager in London, whose working day had just started. The order routing system was producing nonsense. Packages were being assigned to depots in the wrong region. The numbers did not add up.
I was on the data platform team at a logistics software company at the time. We ran a Kafka-based pipeline that ingested order events from half a dozen producer services and fed a routing engine that made real-time dispatch decisions. It was the kind of system that, when it worked, nobody thought about. When it broke, things moved to the wrong place.
What Happened: The Sequence
By 4 a.m. I was in front of my laptop. Consumer lag was normal. No errors in any consumer group. The Kafka brokers looked fine. The routing service logs showed it processing events at the expected rate. From every metric we had, the pipeline was healthy.
What we eventually traced, over the next three hours, was this: one of the producer services had deployed a change roughly 40 hours earlier. A field called destination_zone had been converted from a string enum (e.g., "NW-LONDON") to an integer zone code (e.g., 7). The change was backward-compatible in the Confluent Schema Registry sense because the field was optional in the Avro schema and neither type was registered as the enforced type. The registry had accepted both.
The routing engine received zone as a string, tried to do a lookup by string key, found no match for "7", and silently fell back to a default region. For 36 hours, every order from that producer was being assigned to the default depot. No exception. No error log. Just wrong data, processed quietly, producing wrong outcomes.
The Tooling Gap We Had No Answer For
After the incident we did the usual retrospective. We added tests to the producer. We added a schema compatibility check to the CI pipeline. We wrote a runbook. All correct responses. But the thing that kept bothering me was not that we made the mistake. It was that we had no way to know the mistake was happening.
We had Datadog. We had Confluent's built-in metrics. We had consumer lag dashboards and latency percentile graphs. None of them had any concept of what a valid event looked like in terms of value content and type. The schema registry enforced structural compatibility but had no position on whether a value change was semantically breaking. That distinction matters: the event was structurally valid Avro. It was semantically wrong for our routing logic.
I spent a few weeks after the incident looking at what tools existed for this problem. There were schema evolution utilities, Avro compatibility checkers, DLQ processors for deserialization failures. Nothing that watched a live stream and asked: does this event look like the last hundred thousand events that came through this topic? Is the distribution of this field consistent with what we expect?
The Idea That Would Not Go Away
The concept I kept coming back to was a behavioral baseline. Not a schema registry in the traditional sense. Something that observed real traffic and built a probabilistic model of what normal looked like for each field in each topic. Something that would notice when destination_zone shifted from a distribution of string values to a distribution of small integers, regardless of whether that change was registered anywhere.
This is not a novel idea in the monitoring world. Anomaly detection on time-series metrics has been a standard Datadog feature for years. But nobody had built it for the shape of event data. The distinction is that consumer lag is an aggregate number. Schema drift is about individual field behavior across millions of records. The detection problem is different.
Marcus and I had worked together on a previous project. I described the idea to him over lunch about two months after the incident. We agreed that the hard parts were (1) sampling strategy, because you cannot inspect every field of every event at scale, and (2) sensitivity tuning, because legitimate schema evolution and malicious drift look similar from the outside. He said those sounded like tractable problems and that he would think about them.
Three weeks later he sent me a Jupyter notebook with a prototype field-type inference model running against a synthetic Kafka topic. That was effectively the start of Streamforge.
What We Got Wrong Early
I want to be honest about the wrong turns, because they shaped what we eventually built.
Our first instinct was to make the tool a schema registry replacement. Build the baseline model, register it, enforce it at produce time. We built a prototype connector that could intercept producer output and reject events that deviated from the baseline. It worked in the lab. It did not work in practice because it broke the deployment independence that engineering teams depend on. Producers could not deploy a schema change without coordinating with the monitoring system, which created more friction than the original problem.
We are not saying enforcement at produce time is always wrong. There are contexts where it makes sense, and some of our current integration points allow for it. What we learned is that the primary value proposition is detection on the consumer side, not blocking on the producer side. You want to know that something changed. You want to quarantine the bad events and let the routing engine continue on clean data. You do not want to take the producer offline because it is your one logistics region handling Black Friday traffic.
The second wrong turn was over-investing in alert precision early. We spent three months trying to reduce false positive rates on the anomaly detection before we had enough customers to understand what false positive even meant in context. Different teams have completely different tolerances. A team running analytics pipelines can absorb a 5% false positive rate. A team running financial settlement cannot. You cannot optimize for that without talking to people running real pipelines.
Why the Problem Still Happens in 2025
When we describe Streamforge to engineers, the most common response is some variation of: "yes, we have had this exact incident." Sometimes it is a field type change like ours. Sometimes it is a producer going to null on a previously required field for several hours. Sometimes it is a timestamp field switching from Unix seconds to Unix milliseconds because a developer copy-pasted code from a different service.
These incidents share a common structure: the change is not large enough to trigger a deserialization error, the schema registry does not flag it as incompatible, consumer lag does not spike, and the damage accumulates silently until something downstream is visibly wrong. The discovery mechanism is always reactive: a user complains, a dashboard looks odd, a nightly batch job fails reconciliation.
The reason this keeps happening is not that engineers are careless. It is that the contracts between producers and consumers are implicit. Kafka topics are interfaces, but they are not typed interfaces in the way that function signatures are. There is no compiler that rejects a breaking change. There is just downstream logic that breaks quietly when the assumptions change.
What We Are Building Toward
Two years in, Streamforge does two things that we did not have on that logistics platform: it watches the behavioral baseline of every monitored topic and flags deviations as they happen, and it routes flagged events to a quarantine stream with enough context to understand what changed and why. The routing engine gets clean data. The bad events are not discarded, they are parked with a field-level diff so the team can diagnose and replay once the producer is corrected.
The 3 a.m. incident we started from was a mild one. Forty-eight hours of bad routing data, no permanent loss, recoverable with a backfill. We know teams who have had worse: financial systems booking transactions to incorrect accounts, inventory systems believing stock was present in warehouses it was not, recommendation engines poisoned for weeks by a bad feature value. The common thread is always the same missing layer: something that watches the stream for behavioral change, not just structural validity.
We built that layer because we needed it and it did not exist. That is still the reason we are working on it.