"Self-healing" is one of those terms that gets applied to everything from Kubernetes pods to managed databases, usually meaning "it restarts when it crashes." That's not what we mean when we talk about self-healing pipelines at Streamforge. The problem we're solving is quieter: a pipeline that detects semantically malformed data and makes a routing decision before that data lands somewhere it can't be undone.
This post walks through how the reroute decision tree actually works inside Streamforge, the tradeoffs we made, and where we landed on the question of "how aggressive should automatic intervention be?"
The Reroute Decision Isn't Binary
When Streamforge detects an anomaly on a topic, it doesn't just quarantine the event and move on. It evaluates several signals to decide which of four paths the event should take:
- Pass through: the event flows to the consumer unchanged. Anomaly is logged but not acted on.
- Pass through with annotation: the event reaches the consumer, but a sidecar header is attached with the anomaly metadata. Consumer can inspect or ignore.
- Quarantine: the event is copied to a dead-letter topic. The original consumer does not see it. A notification goes out for human review.
- Block: the event is dropped. Used only when the anomaly confidence is very high and the downstream impact of passing it through is severe (e.g., the event would cause a divide-by-zero or a foreign key violation in a downstream DB write).
The vast majority of schema drift lands in quarantine, not block. Block is reserved for events where we've built high confidence that passing the event causes active damage. This is a conservative default. More on why below.
Signals That Drive the Decision
Three factors push an anomaly toward a more aggressive routing decision:
Anomaly confidence score. This is derived from how far the observed field pattern deviates from the rolling baseline. A field going from 2% null to 55% null over one hour is a high-confidence anomaly. A field shifting from 42% null to 48% null is low-confidence, possibly just natural variance.
Topic criticality tier. When you connect a topic to Streamforge, you assign it a tier: critical (feeds financial systems or user-facing writes), important (feeds analytics and ML features), or standard (feeds internal reporting). Higher tier topics get more aggressive routing defaults. A high-confidence anomaly on a critical-tier topic routes to quarantine automatically. The same anomaly on a standard-tier topic routes to pass-through with annotation.
Downstream impact breadth. Streamforge maintains a lightweight topic dependency graph. If a single upstream topic feeds five downstream consumers, an anomaly on that topic has a multiplied impact. That multiplier shifts the routing decision toward quarantine. An isolated topic with one consumer gets more permissive treatment.
Why We Default Conservative on Blocking
Early in building Streamforge, we spent a lot of time on the question of aggressive automatic remediation. Why not block any high-confidence anomaly automatically? Why make teams opt into block mode per-topic?
The answer is that false positives in block mode cause a different kind of damage than false positives in quarantine mode. If Streamforge incorrectly quarantines a valid event, that event sits in a dead-letter topic, waiting for a human to review and replay it. The pipeline sees a small gap. If Streamforge incorrectly blocks a valid event, that event is gone. In an event-sourcing architecture where the stream is the system of record, that's a meaningful data loss event caused by the monitoring layer itself.
We're not saying aggressive blocking is never the right choice. For specific, narrow anomaly patterns where you have very high confidence and the business logic makes it clearly correct to drop, blocking can be configured per field and per topic. But the default has to be conservative because the failure mode of over-blocking is worse than the failure mode of over-quarantining.
How Quarantine Differs from a Standard DLQ
A standard dead-letter queue in Kafka is a dump for messages that failed to process. It captures the original event and usually a stack trace. What it typically doesn't capture is why the event was quarantined from a schema perspective.
Streamforge's quarantine topics are structured differently. Each quarantined event carries the full field-level diff: which fields deviated, in what direction, by how much compared to the baseline. If you look at a quarantined event in the Streamforge inspector, you see something like:
field: order_total
expected: float (p50=142.80, p99=899.40)
received: string "149.99"
anomaly_confidence: 0.94
first_seen_in_window: 2026-05-09T03:14:22Z
events_affected_in_window: 340That metadata makes replay decisions much faster. The engineer reviewing it knows immediately whether this is a producer bug (wrong type serialization) or a data change (the value legitimately changed format) and can make a replay decision in minutes rather than hours.
The Reroute Latency Constraint
One challenge with inserting a routing decision into the data path is latency. If rerouting adds 200ms to event processing, that's acceptable for batch-adjacent analytics. It's not acceptable for a low-latency pricing pipeline.
Streamforge's reroute logic runs out-of-band by default. The primary consumer receives events at normal speed. Streamforge processes a copy of the event stream in parallel, evaluates anomalies, and writes the routing decision back as a sidecar signal. If Streamforge determines post-fact that an event should have been quarantined, it retroactively marks those events in the quarantine log and sends an alert, but the original consumer is not interrupted.
The inline routing mode, where Streamforge sits in the critical path and actually redirects events before they reach the consumer, is an option for critical-tier topics where the cost of a false-negative (bad data reaching the consumer) outweighs the latency tradeoff. This mode adds roughly 8-15ms of median latency in our testing, which is acceptable for most pipeline configurations but something to evaluate per use case.
Handling Anomalies During Schema Migration Windows
Planned schema migrations are a known challenge for any runtime monitoring system. If your producer team is intentionally rolling out a new field type, you want to update the fingerprint baseline proactively so that Streamforge doesn't quarantine all the "new" events for the duration of the rollout.
The pattern we support is a migration window annotation: an operator declares a planned migration on a topic, which tells Streamforge to shift into "observe and record" mode for the specified fields during the window. Anomalies are logged but routing defaults to pass-through. At the end of the window, if the new patterns have been consistent, the baseline updates automatically to reflect the new normal.
This isn't a bypass of monitoring. It's a structured way to say "we know this is changing, learn from it rather than fire on it." The distinction matters because teams that turn off monitoring entirely during migrations are the teams that don't notice when an unintended change slips in alongside the intended one.
What Happens at the Edges of the Decision Tree
The scenario we haven't solved cleanly yet is correlated multi-field drift. If three fields simultaneously shift in a way that's individually within bounds but collectively indicates a serialization bug, the per-field confidence scores might each be too low to trigger quarantine, while the holistic anomaly is obvious to a human glancing at the event payload.
We're working on a correlation-aware anomaly model that scores joint field deviations, not just individual ones. The challenge is keeping the signal-to-noise ratio acceptable. Joint deviation scoring generates more candidate anomalies, and if the threshold isn't calibrated carefully, it creates exactly the alert fatigue problem we're trying to eliminate.
Self-healing pipelines are not a solved problem. They're a direction. The reroute logic described here covers the 80% case of individual-field drift on well-instrumented topics. The edges still require human judgment, which is exactly what the quarantine layer and the field-level diff inspection are designed to make fast.