Apache Flink's exactly-once processing guarantee is one of its defining features. The documentation explains it clearly: when you enable exactly-once checkpointing, Flink uses a two-phase commit protocol with your source and sink connectors to ensure that each input event contributes to the output exactly once, even across job failures and restarts.
The part that doesn't get enough attention is what "exactly-once" means precisely, and more importantly, where the boundary of that guarantee sits. Exactly-once in Flink means exactly-once with respect to Flink's internal state updates and its committed writes to sinks that support two-phase commit. It does not mean that every downstream consumer of those sinks will see the events exactly once. That distinction has caused real problems in pipelines we've worked on.
What Happens Between Checkpoint and Commit
When a Flink job fails and restarts from its last successful checkpoint, it rolls back its Kafka consumer offsets to the checkpoint position. It then reprocesses all the events between the checkpoint position and wherever the job failed. Events that were processed after the last checkpoint but before the failure are reprocessed.
For the internal Flink state, this is fine. Flink's state management tracks exactly which updates have been checkpointed, so reprocessing doesn't corrupt the state. For writes to Kafka sinks using the Kafka transactional producer (which is what exactly-once mode uses), the uncommitted transactions from the failed run are aborted before the new run starts, so downstream Kafka consumers don't see duplicate writes.
The duplication problem emerges when your Flink job writes to sinks that don't support transactional writes. A Flink job writing to a PostgreSQL database via a standard JDBC sink will, on restart, re-execute all the INSERT or UPDATE statements for the events since the last checkpoint. If those statements aren't idempotent, rows get duplicated. If you're summing counts or amounts in the database, those sums will be inflated.
The At-Least-Once Sink Problem in Exactly-Once Jobs
This is the specific scenario that creates silent data corruption: a Flink job configured with exactly-once checkpointing, reading from Kafka with exactly-once consumer semantics, writing to a non-transactional sink with at-least-once semantics. The Kafka-to-Kafka pipeline is exactly-once. The Kafka-to-database pipeline is at-least-once, disguised as exactly-once.
The team that set up the job sees "exactly-once enabled" in the Flink dashboard and assumes the guarantee applies end-to-end. It doesn't. Flink is transparent about this in the documentation, but it's easy to miss in a fast-moving setup.
Concretely: a Flink job counts user sessions and writes a daily session count per user to PostgreSQL. Job fails at 3pm, restarts from the 2:45pm checkpoint. All session events from 2:45-3:00pm are reprocessed. The session count for each affected user gets incremented again. No error is thrown. The PostgreSQL table now has inflated session counts for users who were active in that 15-minute window.
If your downstream analytics runs at 6pm and queries those counts, it reports higher session numbers than reality. If a downstream ML model trains on session count as a feature, it trains on corrupted data. The failure propagates silently downstream.
Detecting Duplication After a Restart Event
The way we approach this in Streamforge is to treat Flink job restart events as duplication risk signals, not neutral operational events. When we detect a Flink job restart via Flink's metrics endpoint (or via Kafka consumer group offset resets, which are visible even without direct Flink access), we start watching the output topics or output sinks for anomalies consistent with duplication.
The fingerprint for duplication is different from schema drift. You're looking for:
- Event count spikes in a narrow post-restart window, above the rolling baseline for that time period
- Duplicate key values appearing in the output topic within a short time window, when the expected cardinality should be one record per key per window
- Aggregate values in downstream tables that exceed realistic bounds given the source volume
The key field check is often the most reliable signal. If your Flink job is producing one record per user per day as an aggregate, the output topic should have exactly one event per user ID per processing window. Two events for the same user ID within the same window, arriving after a job restart, is a high-confidence duplication indicator.
The Watermark Complication
Event-time processing with watermarks introduces a related edge case. When Flink restarts from checkpoint, it also restores the watermark state. Events that arrived late before the checkpoint may have triggered window completions; those same events, reprocessed after restart, trigger window completions again if the watermark advances past the late event threshold.
This creates a subtler duplication pattern: not duplicate raw events, but duplicate window outputs. A 5-minute aggregation window closes, produces one output record, then Flink restarts, the same window closes again (because events are reprocessed), and produces a second output record. The output record isn't an exact duplicate of the first because the window computation may have seen slightly different events (some late arrivals processed in different order), but both records represent the same logical time window.
Downstream consumers that use upsert semantics to merge window outputs handle this correctly: the second write overwrites the first with the same or similar value. Downstream consumers that use append semantics end up with two rows for the same window.
We're not saying Flink's watermark model is broken. Event-time processing with watermarks is one of the more powerful features in stream processing. The point is that restart behavior with watermarks requires explicit handling in your sink design, and the default append behavior of most JDBC and Kafka sinks doesn't provide that handling automatically.
Making Your Sinks Restart-Safe
The practical fixes fall into three categories:
Use transactional sinks where the sink supports it. The Flink Kafka connector with exactly-once mode handles this correctly via the Kafka transactional producer. The Iceberg connector, the Delta connector, and some database connectors support two-phase commit. If your sink has a transactional mode, use it.
Make your JDBC writes idempotent. If you're writing to PostgreSQL, design your queries as UPSERT on a stable key rather than INSERT. If a row with that key already exists, the upsert overwrites it with the same value rather than creating a duplicate. This requires that your output records have a stable, deterministic key that represents the logical unit of output, not just an auto-increment ID.
Track checkpoint IDs in your output. For sinks where neither of the above is feasible, embed the Flink checkpoint ID in the output record as a column. Write a post-restart deduplication job that flags any records with the same logical key but different checkpoint IDs as candidate duplicates. This is the most operationally expensive approach and should be the fallback, not the default.
The Detection Loop
The reason this problem persists is that duplication after restart is often attributed to something else. A session count that's 3% higher than expected after a maintenance window might be chalked up to normal variance. A doubled row in a table might be found weeks later by a data analyst who assumes it's a data quality issue in the source system.
Correlating output anomalies with Flink restart events closes that attribution loop. When Streamforge surfaces a duplication signal and timestamps it to a job restart event 12 minutes prior, the on-call engineer has a much clearer hypothesis to investigate than "something seems off with the user session counts."