The Consumer Was Down for Eleven Days and the Topic Retains Seven: Rebuilding the Offset Gap That Kafka Retention Silently Ate

Eleven days down. Seven days retained. The arithmetic is the whole incident, and it is not a Kafka problem. It is an offset-gap problem, and the gap is permanent the moment the committed offset falls below the partition’s low watermark.

This article is about that moment: how to detect it in one query, how to tell it apart from a silent rewind that looks identical on a lag graph, and how to price the rebuild honestly. The eleven-and-seven numbers are the scenario, not a case record. The mechanics below are documented and reproducible on your own cluster.

The failure signature is a vertical line, not a slope

A slow consumer produces a backlog that ramps up at a finite slope. A consumer that has lost its position produces a near-instant jump to multi-day lag. The Karafka documentation on offset rewind states the distinction plainly: a near-instant jump to multi-day lag is only physically possible if the consumer’s offset pointer has been moved backward to an older position.

When the consumer restarts after eleven days, it asks the broker for its committed offset. The broker answers with offset_out_of_range because that offset has been trimmed. The client then applies auto.offset.reset. If that setting is earliest — the common default in many client configurations — the consumer silently seeks to the oldest retained offset and begins consuming from there. The lag graph shows a vertical rise to the retention ceiling, then a slow drain. Nothing pages. Nothing errors. The pipeline is now missing four days of events and will never know unless someone checks.

The Karafka documentation describes this exact path as expected behavior, not a bug: consumer downtime longer than the retention window causes Kafka to legitimately return offset_out_of_range, and the client resets to earliest.

Retention is a property of the log, not a promise about your position

The default cleanup policy is delete, which discards old log segments once their retention time or size limit is reached, per the Confluent topic configuration reference. The default retention is 168 hours, or seven days, per Confluent’s retention guide. Both can be overridden per topic with kafka-configs.sh --alter --entity-type topics --entity-name <topic> --add-config retention.ms=....

Two mechanics make “seven days retained” a weaker guarantee than it sounds.

First, deletion happens at segment granularity. A message past its retention age is not removed until its entire segment is eligible. The AutoMQ retention wiki puts it directly: when a message’s age exceeds the configured retention period, the entire segment containing that message becomes eligible for deletion, but only if all messages within that segment have also exceeded the threshold. The active segment is never cleaned up regardless of retention settings. So the actual earliest available offset is a function of segment.ms, segment.bytes, and write rate — not just retention.ms.

Second, retention and offset retention are two separate clocks. A consumer group inactive beyond offsets.retention.minutes (default seven days, per the AutoMQ wiki) loses its committed offsets entirely. When it reactivates, it has no position to fall back to and starts from auto.offset.reset. This failure mode looks identical on the lag graph but has a much cheaper fix: the messages are still in the log, and the position can be reconstructed from a known timestamp or a downstream checkpoint.

An eleven-day outage can trip both clocks. That is why the first diagnostic question is not “why did the consumer stop” but “did the offset move backward, or did the log move forward past it.”

One query classifies the failure

When offset_out_of_range appears, the broker includes the partition’s low and high watermarks alongside the requested offset. The comparison tells you which sub-case you are in.

• Requested offset below the low watermark: the consumer’s position is below the oldest retained offset. This is a genuine retention overrun or a client-side rewind. The front of the log is gone.
• Requested offset above the high watermark: the committed offset is ahead of the log end. This indicates broker-side data loss — unclean leader election, log truncation. Investigate broker log integrity before anything else.

You can reproduce the watermark check without waiting for an error. Use kafka-consumer-groups.sh --describe to get the group’s committed offset per partition, and kafka-get-offsets.sh (or GetOffsetShell) to get the earliest and latest offsets. If the committed offset is less than the earliest offset, the gap is real and no retention setting recovers it.

Before running any reset, note the operational constraint: resetting consumer offsets requires first emptying the consumer group by stopping its clients or consumers, per the Confluent Cloud offset-reset documentation. A reset against a live group either fails or produces undefined behavior. Plan the stop window.

Two paths to the same graph, two different fixes

The lag graph cannot distinguish a retention overrun from a silent rewind. The watermark check can, and the fix differs by an order of magnitude in cost.

Retention overrun. The consumer was down longer than the topic retains. The missing window must be rebuilt from upstream. Kafka has nothing to give you.

Silent rewind. The consumer’s offset moved backward while the messages were still in the log. The Karafka documentation catalogs the known causes: librdkafka cooperative-sticky bugs below 2.4.0, broker-side segment races in Kafka versions before 0.10.0 and in 2.4.0–2.5.0, and manual operator resets. The distinguishing signal is timing: rewind bugs correlate with rebalance events or segment rolls, not with the consumer’s downtime. If the consumer was up and the lag still spiked to the retention ceiling, you are debugging a client or broker bug, not paying for a backfill.

This distinction matters because the rewind case is cheap. Seek the group to the correct position and replay. The overrun case is expensive. You are about to reconstruct days of change data from a source system that may not have a clean snapshot boundary.

Price the gap in reconciliation work, not in Kafka terms

The cost of a retention overrun scales with the number of downstream tables that consumed the topic, not with the topic’s size. A CDC topic feeding one warehouse table is a different job from the same topic feeding forty dbt models across three marts.

Consider an illustrative case — hypothetical, not a recorded incident. A CDC topic with seven-day retention feeds a warehouse. The consumer is down eleven days. Four days of messages are gone. The recovery has two parts:

1. Snapshot at the retention boundary. Take a consistent snapshot of the source tables as of day seven, the last point Kafka can still replay from. This is a source-system operation, not a Kafka one. Its cost is the snapshot size and the lock or replication-slot overhead it imposes on the source.
2. Replay days seven through eleven from Kafka. This is the part Kafka can still do. It requires the consumer to start from the earliest retained offset and apply events idempotently, because the snapshot and the replay will overlap at the boundary.

Then reconcile. Count rows and distinct keys per table between the rebuilt warehouse state and the source. Any mismatch is either a missed event or a duplicate applied twice. The reconciliation query is cheap to write and expensive to run at scale, and it is the only honest proof that the gap is closed.

The on-call hours are dominated by the snapshot coordination and the reconciliation, not by the Kafka replay. Budget accordingly.

The upstream side has its own retention clock

If the topic is fed by Postgres logical decoding, the outage did not only affect Kafka. A replication slot persists across crashes and knows nothing about the state of its consumer, per the PostgreSQL logical decoding documentation. It will prevent removal of required WAL even when no client is connected. An eleven-day consumer outage means eleven days of WAL retained on the primary, and in extreme cases the database can shut down to prevent transaction ID wraparound.

This changes the recovery plan. You cannot simply restart the CDC connector and expect it to resume from where it stopped if the slot was dropped to reclaim disk. And if the slot was not dropped, the WAL volume is the first thing to check before you add replay load.

The PostgreSQL documentation also notes that a slot’s position is persisted only at checkpoint, so after a crash it may return to an earlier LSN and resend recent changes. Any replay path must be idempotent on the consumer side. This is not optional hygiene; it is a documented behavior of the slot.

A recovery runbook that does not pretend the gap is smaller than it is

The sequence below assumes a confirmed retention overrun: committed offset below the low watermark, messages gone. Adjust if the watermark check says otherwise.

1. Stop the silent reset. Set auto.offset.reset=error on the consumer before restarting it. This converts a silent seek-to-earliest into a stopped partition with a raised error. The Karafka documentation is explicit that this is a diagnostic and a guard, not a mitigation: it only fires when the consumer has no valid position to fall back to. It does not preserve a position, because there is none to preserve. Do not enable it without alerting on the resulting error and a runbook to seek the partition, or the guard becomes the outage.

2. Record the watermarks. Capture the committed offset, the low watermark, and the high watermark per partition before any reset. This is the evidence that classifies the failure and the baseline for reconciliation.

3. Empty the group. Stop all consumers in the group. Offset reset requires an inactive group, per the Confluent Cloud documentation. For groups using manual assign(), locate and pause the assigning consumer — such groups appear empty in tooling but may hide running consumers.

4. Seek to the earliest retained offset. Reset the group to earliest. This is the start of the replayable window, not the start of the gap. The gap is everything before it.

5. Snapshot the source at the boundary. Take a consistent snapshot of the upstream tables as of the timestamp corresponding to the earliest retained offset. This is the expensive step. Its duration depends on source size and the snapshot mechanism available.

6. Replay and apply idempotently. Consume from earliest with deduplication on a stable key. The snapshot and the replay overlap at the boundary; the consumer must tolerate seeing the same change twice.

7. Reconcile. Compare row counts and distinct keys per downstream table against the source. Document any residual gap explicitly. An undocumented gap is worse than a documented one.

What the guard costs and what it does not buy

auto.offset.reset=error is the cheapest change in this article and the one most likely to be deployed without its companion. Alone, it converts silent data loss into a stopped partition. That is a better failure mode, but it is still a failure. The companion is an alert on the offset-reset error and a runbook that tells the on-call engineer which partitions to seek and to what position. Without that, the guard pages someone at 3 a.m. with no action to take.

The setting does not prevent retention overrun. It does not extend the replay window. It does not recover the gap. It only makes the gap visible at the moment it happens instead of weeks later during a reconciliation that does not tie out.

The maintenance tax is in the configuration, not the incident

Retention settings, offset retention, and rebalance behavior are configuration decisions that determine whether an outage is a two-hour catch-up or an eleven-day rebuild. They are usually set once, at topic creation, and never revisited.

Three checks are worth running before the next outage, not after.

Compare retention to realistic downtime. If the topic retains seven days and the consumer has no documented recovery-time objective shorter than that, the retention window is a guess. The default is 168 hours because it is a default, not because it matches your recovery posture.

Check offsets.retention.minutes separately. A consumer down past the offset-retention window loses its position even if the messages are still in the log. That failure is cheaper to fix but looks identical on the graph. Knowing which clock expired first saves hours.

Know your rebalance behavior. If the client library is librdkafka below 2.4.0 and uses cooperative-sticky assignment, the rewind bug is a live possibility. The Karafka documentation notes that switching to an eager assignor eliminates the implicit-resume path, at the cost of pausing the entire group on every rebalance. That is a tradeoff, not a fix. The fix is the upgrade.

None of these checks prevents an eleven-day outage. They determine whether the recovery is a replay or a rebuild. That is the maintenance tax: the work you do before the incident decides which invoice you pay after it.

FAQ

Does increasing retention now recover the missing days? No. Retention governs future deletion. Messages already deleted are gone. Increasing retention.ms changes the window for events written from now on, not for events written eleven days ago.

Can I reset the offset to a timestamp instead of earliest? Yes, kafka-consumer-groups.sh --reset-offsets --to-datetime seeks to the nearest offset at or after a timestamp. But if the timestamp falls before the earliest retained offset, the reset lands at earliest anyway. The watermark check tells you whether the timestamp is inside the replayable window.

Why did the consumer not error when it reset? Because auto.offset.reset defaults to a silent seek in many client configurations. The broker returned offset_out_of_range; the client handled it by seeking. Setting the value to error changes the client’s response, not the broker’s.

Is tiered or remote storage a fix? It changes the economics of long retention by moving old segments off broker-local disk, but it does not change the offset-gap mechanics. If the committed offset is below the low watermark of the tiered log, the gap is still a gap. Tiering extends the window; it does not guarantee your consumer’s position is inside it.

How do I know the rebuild is complete? Row counts and distinct keys per downstream table match the source at the reconciliation timestamp. Anything short of that is a documented gap, not a recovery.