Kinesis Data Streams is a solid choice for high-throughput event ingestion in AWS-native architectures. The managed infrastructure, the native integration with Lambda and Firehose, and the predictable per-shard pricing make it appealing for teams that don't want to operate Kafka. But Kinesis has a few sharp edges that don't announce themselves with errors. They just drop data quietly, and your downstream consumers have no idea.
We've integrated Streamforge with Kinesis pipelines and worked through all three of the patterns described here. None of them are AWS bugs. They're documented behaviors that interact badly with common consumer implementations. Knowing them in advance changes how you build and monitor.
Pattern 1: Shard Iterator Expiration
Kinesis uses shard iterators to track your consumer's position in a shard. An iterator is valid for 5 minutes. If your consumer doesn't call GetRecords within 5 minutes of getting an iterator, the iterator expires and you need to request a new one.
The problem is how many consumers handle iterator expiration: they request a new iterator with LATEST, not with the sequence number they were last at. LATEST gives you a position at the current head of the shard, not where you stopped reading. Everything between your last successful read and the new LATEST position is silently skipped.
This happens most often when a consumer pauses due to backpressure from a downstream system. The consumer is working through a batch, the downstream database is slow, the consumer stops calling The fix is to handle iterator expiration by re-requesting from the last processed sequence number, not from The monitoring signal for this pattern: if you're tracking sequence numbers consumed versus the sequence numbers being produced, a gap during a period of consumer slowdown is the fingerprint. Streamforge detects this by sampling the sequence number distribution over time and flagging discontinuities that don't correlate with a shard split event. Each Kinesis shard handles up to 1,000 records per second or 1 MB/s for writes, whichever is lower. When you exceed these limits, the Kinesis API returns a The silent data loss occurs when the retry budget is exhausted. If a producer is in a sustained burst mode and shard capacity isn't sufficient, retries keep failing. After the maximum retry attempts, the SDK typically logs the failure and discards the event. The produce call returns successfully from the application's perspective because the retry loop completed. The event is gone. This pattern is particularly insidious because it's traffic-correlated. You'll see it during peak load periods, precisely when you most need your pipeline working. The producer metrics look healthy: throughput is up, the application is producing. But the events that couldn't fit in the shard capacity window are quietly dropped. The right fix has two parts. First, instrument your producer to track We're not saying Kinesis's capacity model is broken. It's a predictable model that you can engineer around. The mistake is assuming the retry logic in your SDK will always succeed without monitoring the cases where it doesn't. When you scale a Kinesis stream by splitting shards, the original parent shard transitions to CLOSED state. Records that were written to the parent shard before the split remain readable from the parent for 24 hours (or up to 7 days with extended retention). After that, they're gone. Consumer implementations that don't handle shard topology changes correctly can skip the parent shard's tail. When the consumer's shard discovery logic notices the new child shards, it might start consuming from the child shards immediately without exhausting the parent. Everything in the parent shard after the split is never read. This doesn't happen in well-maintained Kinesis consumer libraries like KCL (Kinesis Client Library), which tracks parent-child shard relationships and ensures parent shards are drained before child shards are consumed. But custom consumers or older SDK integrations often don't implement this correctly. The scenario we encountered most recently: a media analytics pipeline built on a Lambda function that polled shards via EventSourceMapping. The team manually triggered a shard split during a traffic spike. Lambda's Kinesis integration handled the new child shards correctly but didn't drain the parent before starting on the children. Roughly 40 minutes of clickstream events from the parent shard's tail were skipped. The downstream analytics table showed a dip in event volume for that window that was initially attributed to a traffic drop, not to data loss. The difficult scenario is when two of these patterns coincide. A shard split during a traffic spike (pattern 3) often occurs alongside producer throttling (pattern 2) because the traffic spike is what drove the split decision. Your consumer is trying to adapt to the new shard topology while producers are hitting capacity limits. In that window, you can have both events failing to produce (due to throttling retry exhaustion) and events failing to consume (due to parent shard draining failure). The standard CloudWatch metrics for Kinesis don't surface either pattern directly. The metrics we've found most useful for detecting these patterns are derived rather than native. Derived metrics require you to instrument at the application level: Streamforge integrates with Kinesis pipelines by consuming a mirrored copy of the stream and comparing the sequence number series against what downstream consumers report as processed. When there's a discrepancy, the alert identifies which pattern it matches and what time window is affected. That gives the on-call engineer enough context to determine whether data recovery is possible (within the retention window) or the gap is permanent. Kinesis is reliable infrastructure. The data loss patterns above are predictable and preventable. They just require knowing they exist before you hit them at 2am.GetRecords for 6 minutes, iterator expires, consumer requests LATEST
LATEST. This requires your consumer to track and persist the last sequence number, which adds implementation complexity. Most consumer SDKs handle this correctly if you configure them properly, but the default behavior in raw API usage is to fall back to LATEST.Pattern 2: Producer Throttling Without Backpressure Feedback
ProvisionedThroughputExceededException. If your producer SDK is configured to retry on this exception (which most are, out of the box), events are retried with exponential backoff.ProvisionedThroughputExceededException counts explicitly, not just overall error rates. Second, size your shard count for peak load rather than average load. Kinesis shard scaling (via UpdateShardCount) requires an explicit API call, so you need to know in advance when bursts are coming, or build auto-scaling logic that responds quickly enough.Pattern 3: GetRecords Iterator Gaps After Shard Splits and Merges
How These Patterns Interact
GetRecords.IteratorAgeMilliseconds tracks how far behind your consumer is relative to the latest records, but it doesn't show sequence number gaps. WriteProvisionedThroughputExceeded tracks throttle events but doesn't tell you how many events were ultimately dropped after retry exhaustion.What to Monitor