Schema evolution has a formal definition in the context of Avro: a schema change is backward compatible if a consumer using the old schema can read events written with the new schema. It is forward compatible if a consumer using the new schema can read events written with the old schema. It is fully compatible if both directions work.
In practice, most teams focus on backward compatibility because consumers are updated less frequently than producers and old events need to remain readable. The rules are not complicated, but producers break them regularly, usually without realizing the downstream impact until something downstream fails.
The Three Changes That Cause Most Problems
After looking at schema-related incidents across the pipelines we monitor, three types of changes cause the majority of the problems we see.
Adding a Required Field Without a Default
The most common mistake is adding a new field to an Avro schema as required (without a default value) and registering it as a new schema version. From the Avro specification perspective, this is a backward-incompatible change: a consumer reading with the old schema does not know about the new field and has no default to fall back on. The Confluent Schema Registry with backward compatibility mode enabled will correctly reject this.
Where it slips through is when teams use compatibility mode "NONE" (turned off entirely, which is the default in some environments) or when they use Protobuf with implicit optional semantics and do not model required fields explicitly. The change looks safe during testing because the test consumer uses the new schema. It only breaks when the old consumer version (which was not updated as part of the rollout, perhaps because it lives in a different service with a different deployment cycle) tries to process the new events.
The safe pattern: any new field added to a schema should have a default value. In Avro, this means the field must be defined as a union with null as the first type (making it nullable) or as a type with an explicit default. In Protobuf, fields in proto3 are always optional with zero-value defaults, so this is less of a problem, but teams that rely on proto3 default behavior for semantic meaning rather than optional presence can still create implicit required fields in their application logic.
Renaming a Field
Renaming a field is always a breaking change for backward compatibility in Avro. The old schema has user_id. The new schema has userId. An older consumer looking for user_id finds nothing. The field resolution in Avro uses aliases, and if aliases are properly set, an old consumer can find the renamed field. In practice, aliases are rarely set because they require knowing in advance that you will rename the field and proactively documenting the alias in the original schema.
Teams rename fields for a variety of reasons: standardizing naming conventions (snake_case to camelCase migrations are a common trigger), fixing typos that slipped into production, or aligning with a new domain model. Each of these is reasonable in isolation. The problem is that in a Kafka topic, the old events with the old field name remain in the partition log for as long as the retention period. A consumer replaying from the beginning of the log will see the old field name for events before the rename and the new field name for events after. If the consumer does not handle this dual-name period, replay fails.
The safe pattern: instead of renaming, add the new field alongside the old one. Keep both for a migration period long enough to ensure all consumers have been updated to use the new field. Deprecate and then remove the old field only after confirming no active consumer group still reads it.
Changing a Field's Type
Type changes are the most dangerous because they often appear innocuous and produce no structural error, just wrong data. Changing a field from string to integer (or vice versa) is clearly breaking. But the subtle version is changing a numeric type within a type family: from int32 to int64 (usually safe but can break consumers using strict type binding), from float to double (precision expansion, sometimes breaks strict equality checks), or from decimal to float (precision loss that may not be immediately visible).
The version we see most often causing real damage is unit changes that look like type-compatible changes: a timestamp field changing from Unix seconds to Unix milliseconds. Both are integer values. Both pass schema validation. The new values are three orders of magnitude larger. A consumer computing time deltas suddenly gets results in the billions of seconds range. A consumer formatting dates gets results in 2001 instead of 2025 because it interprets milliseconds as seconds.
We are not saying all type changes are necessarily wrong. Sometimes you genuinely need to upgrade a type (int to long is a common and often necessary evolution). The key is making the change visible and coordinating with consumer teams before it hits production, not after.
Where the Schema Registry Helps and Where It Does Not
The Confluent Schema Registry, with backward compatibility mode enabled, will reject the first and third patterns (adding a required field and changing field types in incompatible ways). It does not catch the unit-change case (milliseconds vs seconds) because from the schema's perspective both are the same type. It does not catch renaming unless aliases are properly configured. And it only enforces rules if compatibility mode is enabled and if the producer is using the schema registry integration, neither of which is universally true.
Schema registry enforcement is a useful first layer. It prevents the hard structural violations. The behavioral violations, including unit changes, semantic field repurposing, value range changes, and null rate changes, pass through schema validation without complaint. Those require a different approach: behavioral monitoring of the live stream rather than structural validation at produce time.
The Backward Compatibility Rules in Plain Language
For teams managing Avro schemas on Kafka topics, the practical rules for backward-compatible evolution are:
- Adding a field: always provide a default value (null or a sensible domain default)
- Removing a field: mark as deprecated first, keep for one full consumer rotation, then remove
- Renaming a field: add the new name alongside the old, run both in parallel, remove old after migration
- Changing a type: treat as a breaking change unless you can verify all active consumers handle both types, which usually means a versioned topic or a parallel migration period
- Changing value semantics: the schema registry will not catch this; coordinate explicitly with consumer teams and add behavioral monitoring to detect the change at runtime
How Streamforge Monitors for These Violations
The behavioral baseline Streamforge builds for each topic tracks field type distribution, null rate per field, and value range per numeric field. When a type change or unit change lands in the stream, the field's value distribution shifts outside the baseline's normal range and triggers an anomaly alert before the downstream consumer processes significant volume of the bad events.
For the milliseconds-vs-seconds case, the detection is straightforward: the baseline has learned that a timestamp field contains values in a specific decade-level range (Unix seconds for 2023-2025 are in the range 1.6 to 1.8 billion). When values jump to the 1.6 to 1.8 trillion range, the anomaly fires within the first few hundred events.
Contract tests and schema registry enforcement should be your first line of defense because they prevent problems before they reach production. Behavioral monitoring is the catch-all for the changes that slip through, which, based on the patterns we observe, is not rare. It is a regular occurrence on any active codebase with multiple teams contributing to producer services.