Streamforge Documentation

Connect your first pipeline in 15 minutes. No code changes to producers or consumers.

Where to start

If you're new to Streamforge, start with the Quickstart guide. It walks through connecting a Kafka broker, selecting topics, and triggering a test schema detection in about 15 minutes.

Schema Guard Configuration

Schema Guard validates incoming events against a learned baseline per topic. Configuration is per-topic and can be changed without restarting monitoring.

# schema-guard config for a topic
topic: orders-v2.checkout
schema_guard:
  mode: quarantine  # warn | quarantine | block
  strictness: standard  # standard | strict
  null_threshold: 0.01  # flag if null% exceeds baseline by this factor

Rule types

• warn: log the violation, send an alert, do not quarantine the event.
• quarantine: redirect the event to a dead-letter stream for inspection and optional replay.
• block: drop the event and send an alert. Use only for strict compliance requirements.

Self-Heal Routing Rules

Routing rules define what happens when Schema Guard triggers. Rules are evaluated in order; the first matching rule applies.

routing_rules:
  - match: null_violation
    action: quarantine
    destination: orders-v2.checkout.dlq
  - match: type_coercion
    action: warn
  - match: volume_drop
    threshold_percent: 50
    action: alert

Event Replay

After a producer is fixed, you can replay quarantined events back into the main topic using the DLQ Inspector or via API. Replay preserves the original partition and ordering within each partition.

DLQ Inspector

The DLQ Inspector shows all quarantined events with a full schema diff: the expected value type/distribution vs what was received. You can filter by topic, violation type, or time range, then replay or dismiss in bulk.

Alerts and Webhooks

Configure alert destinations in your workspace settings. Supported: Slack webhook, PagerDuty routing key, OpsGenie API key, SMTP email, or a custom webhook URL.

{
  "alert_channels": [
    {
      "type": "slack",
      "webhook_url": "https://hooks.slack.com/services/...",
      "dedup_window_minutes": 15
    }
  ]
}

See the API Reference for the full list of endpoints to manage alert channels programmatically.