Schema Guard, Self-Heal Routing, and DLQ Inspector
Three focused modules for one problem: silent data loss in event-driven pipelines. Not a general observability platform. Not a schema registry replacement. The layer that watches what your producers are actually sending and acts before your consumers see the damage.
Schema Guard
Field-level schema validation that learns from your real traffic. No manual schema registry required, no Avro or Protobuf prerequisite. Streamforge builds a probabilistic model per topic from observed events: field names, inferred types, nullability percentiles, and value cardinalities. When a new event deviates from that model, it flags the specific field and the nature of the deviation.
Configurable strictness per topic: warn-only for low-risk streams, auto-quarantine for financial or user-facing pipelines, or block to hard-stop bad events from reaching consumers.
- Baseline built from first 10,000 events per topic (configurable)
- Rolling 30-day window distinguishes intentional evolution from drift
- Catches: null field creep, type coercions, unexpected key additions, value-range anomalies
- Works on JSON, Avro, Protobuf, and binary topics
| Topic | Trigger | Action | State |
|---|---|---|---|
| orders-v2.checkout | null event_id | quarantine | ACTIVE |
| payments.events | type coercion | warn | RESOLVED |
| inventory.updates | volume drop | alert | OK |
Self-Heal Routing
Define reroute rules per topic: breach threshold triggers automatic redirect to a quarantine stream. Your consumers never see the bad events.
When the issue resolves, replay from quarantine in order. Works with Kafka, Kinesis, and Flink without modifying your broker configuration.
- Unlimited reroute rules on paid tiers
- Sub-60s response target on Scale tier
- Bulk replay up to 500K events once producer is fixed
DLQ Inspector
Every quarantined event comes with a full schema diff: what field violated, the expected value distribution, the actual value received, and the producer partition/offset. Filter, replay, or dismiss events in bulk.
| Field | Expected Type | Actual Value | Violation |
|---|---|---|---|
| order_id | string(uuid) | ord_3a71c2... | PASS |
| event_id | string(required) | null | NULL VIOLATION |
| amount | float64 | 142.50 | PASS |
| currency | string(iso-4217) | 840 | TYPE COERCION |
Alerts that respect your on-call
One alert per incident, not one per failed message. Configurable dedup window from 1 minute to 4 hours. Routes to PagerDuty, Slack, OpsGenie, or email. Suppression schedules for maintenance windows.
What Streamforge is not
Knowing what a tool doesn't do is as important as what it does.
Not a schema registry
Confluent Schema Registry enforces registered schemas at produce time and works well for Avro/Protobuf shops. Streamforge works on any topic format (JSON, binary, mixed) by learning from live traffic. The two tools are complementary, not competitive. We fill gaps the registry can't see.
Not a general observability platform
Datadog and Grafana give you infrastructure metrics: CPU, memory, consumer lag by partition. Streamforge focuses on the data itself: what fields are in each event, what types they carry, whether the schema contract between producer and consumer is holding. Different problem, different layer.
Not a data quality platform
Monte Carlo and Great Expectations catch data quality issues in your warehouse after the fact. Streamforge catches anomalies in the stream, before they reach the warehouse. The goal is to stop bad data at the pipeline layer, not clean it up downstream.
Not a stream processing engine
We don't transform, aggregate, or enrich your events. We watch them and act when something breaks the expected schema. Your Flink jobs and Kafka Streams applications continue doing what they do. Streamforge sits alongside them, not inside them.
Connect your first pipeline in 15 minutes.
No credit card required. Free tier includes Schema Guard and DLQ Inspector for one pipeline.