The CDC Consumer Was Paused for a Long Weekend and the Primary’s Disk Filled With WAL: Sizing max_slot_wal_keep_size Before It Sizes You

A logical replication slot does not care whether your CDC consumer is running. It persists across crashes, it knows nothing about the state of the process reading from it, and it will hold back WAL removal for as long as it exists. The PostgreSQL documentation states this plainly: slots “will prevent removal of required resources even when there is no connection using them.”

That sentence is the whole incident. A connector goes quiet on Friday. Nobody notices because the connector’s own health check is green — it is up, it is just not consuming. By Monday the primary’s pg_wal volume is full, checkpoints cannot complete, and the database stops accepting writes. The failure is not in the connector. It is in a retention decision nobody made.

What the cap actually does, and when

max_slot_wal_keep_size is documented as the maximum size of WAL files that replication slots are allowed to retain in pg_wal at checkpoint time. Two words in that sentence matter more than the rest: checkpoint time. The cap is not enforced continuously. It is evaluated when the checkpointer runs, which means the gap between “slot crossed the limit” and “slot is invalidated” is bounded by your checkpoint interval, not by your alerting.

The default is -1, and the documentation is explicit about what that means: replication slots may retain an unlimited amount of WAL. There is no backstop. The only thing standing between a paused consumer and a full disk is the size of the disk.

When the cap is set to a non-negative value and a slot’s restart_lsn falls behind the current LSN by more than that size, the documentation says the standby using the slot “may no longer be able to continue replication due to removal of required WAL files.” The slot is not gracefully paused. The WAL it needed is gone, and the consumer cannot resume from where it stopped.

This setting can only be set in postgresql.conf or on the server command line. It cannot be changed from a SQL session with SET, but editing postgresql.conf and reloading the configuration applies the new value without a restart. It is not a knob you can flip from a session during an incident, and it is not something you discover you needed at 2 a.m.

Read the slot view as a countdown, not a dashboard

pg_replication_slots exposes the state you need before the disk fills. The columns that matter:

• restart_lsn — the oldest WAL still possibly required by the consumer. The distance between this and pg_current_wal_lsn() is the retention you are paying for.
• wal_status — reserved means the claimed files are within max_wal_size; extended means max_wal_size is exceeded but files are still retained by the slot or by wal_keep_size; unreserved means the slot no longer retains the required WAL and some will be removed at the next checkpoint; lost means the slot is no longer usable.
• safe_wal_size — the number of bytes that can still be written to WAL before the slot is in danger of entering lost. It is NULL for lost slots and NULL when max_slot_wal_keep_size is -1.
• inactive_since — when the slot stopped being streamed. This is the timestamp that tells you how long the consumer has actually been gone, as opposed to how long your monitoring has been confused.
• invalidation_reason — wal_removed, rows_removed, wal_level_insufficient, or idle_timeout. If this is non-NULL, the slot is already dead and you are in recovery, not prevention.

A runnable diagnostic that turns the view into a countdown:

SELECT
  slot_name,
  active,
  wal_status,
  inactive_since,
  pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
  ) AS retained_wal,
  pg_size_pretty(safe_wal_size) AS safe_wal_size,
  invalidation_reason
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

If safe_wal_size is NULL and max_slot_wal_keep_size is -1, you have no countdown at all. That is the configuration most clusters ship with, and it is the one that produces the Monday morning outage.

Alerting on disk percentage alone is too late. By the time pg_wal is at 90%, the remaining headroom may be less than one checkpoint interval of write traffic. Alert on safe_wal_size shrinking, and on inactive_since aging past your expected consumer restart window.

Two failure modes, priced differently

There is no setting that eliminates risk. There is a choice about which risk you carry, and the two are not symmetric.

Failure mode A: no cap

With max_slot_wal_keep_size = -1, retention is bounded only by disk. The operational write-up from Netdata describes the sequence: WAL accumulates, the volume fills, checkpoints cannot complete, and PostgreSQL eventually shuts down. Recovery is not a matter of dropping the slot and moving on. If the disk is already 100% full, dropping the slot may still require a checkpoint before space is freed, so you need emergency disk space before anything recycles.

And you cannot delete files from pg_wal to buy room. The same guide is blunt: doing so corrupts the database and breaks replication. Segments are recycled only after a checkpoint and only when no consumer needs them. The primary’s availability is the thing at risk, and the recovery path runs through infrastructure you may not control at 3 a.m.

Cost shape: unbounded, and it lands on the primary. Every writer on that database is affected, not just the CDC pipeline.

Failure mode B: cap set too low

With a cap in place, the slot is invalidated when the consumer falls too far behind. Gunnar Morling’s write-up on replication slots describes the consequence for a Debezium-style connector: the slot becomes unusable, a new slot must be created, and that typically means a fresh initial snapshot of the data.

Cost shape: bounded, and it lands on the consumer. You pay for a re-snapshot — read load on the primary, time proportional to table size, and downstream reprocessing of everything the consumer missed. You do not lose the primary.

That asymmetry is the argument for setting the cap. A capped slot converts an availability incident on the system of record into a data-pipeline incident on a system you can rebuild. The rebuild is expensive. The alternative is worse.

Sizing procedure: arithmetic, not vibes

The cap should be chosen when you know two numbers: how fast the primary generates WAL, and how long the consumer might plausibly be gone. Neither is a guess you make during an incident.

Step 1 — measure WAL generation over a representative window. Sample the current LSN at two points and divide by elapsed time. Run this across a busy period, not an idle afternoon:

SELECT pg_current_wal_lsn(), now();
-- wait a representative interval, then:
SELECT pg_current_wal_lsn(), now();

SELECT pg_size_pretty(
  pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0'::pg_lsn)
) AS total_wal_since_start;

For a rate, take the difference between two samples and divide by the seconds between them. Do this on a weekday peak and on a batch-load night; the peak is what sizes the cap.

Step 2 — multiply by the longest plausible consumer outage. Not the average restart time. The holiday weekend. The on-call handoff gap. The connector upgrade window that ran long. This is a policy decision disguised as a number, and it should be written down next to the setting.

Step 3 — add headroom for everything else that retains WAL. The cap is not the only consumer of pg_wal. The WAL configuration documentation notes that a slow or failed standby using a replication slot causes old WAL files to accumulate, and that max_wal_size is never a hard limit, so headroom should always be left to avoid running out of disk space. Archiving that falls behind retains segments too. Your cap has to coexist with all of it on the same volume.

Step 4 — compare to free space and decide. If the computed retention exceeds the free space on the pg_wal volume, you have two options: grow the volume, or lower the tolerated outage window and accept more frequent re-snapshots. There is no third option where the disk is bigger than it is.

A worked hypothetical, with placeholder numbers that are not measurements from any system: suppose the primary generates 2 GB of WAL per hour at peak. The longest tolerated consumer outage is 72 hours. That is 144 GB of retention before headroom. Add 30% for checkpoint and archive retention and you are at roughly 187 GB. If the pg_wal volume has 200 GB free, the cap is set at the edge and any archive backlog pushes you over. If it has 400 GB free, you can set the cap at 144 GB and still have room for everything else. The numbers are yours to fill in; the arithmetic is the point.

The adjacent levers that change the arithmetic

The cap is one control. Three others change what the cap needs to be.

idle_replication_slot_timeout invalidates slots that have been inactive longer than the configured duration. The documentation notes that invalidation occurs at checkpoint, so there is lag between the timeout being exceeded and the slot being invalidated, and that a value of zero disables the mechanism. This is the right tool for slots that are genuinely abandoned — a decommissioned connector, a replica that was never rebuilt. It is the wrong tool for a consumer that is expected back, because it will kill the slot on a schedule rather than on a decision.

Heartbeats address a different problem: a slot on a low-traffic database that never receives change events and therefore never advances. Morling describes the mechanism — a slot for an idle database cannot make progress because it never receives changes, so it retains more and more WAL. PostgreSQL 14 and later provides pg_logical_emit_message() to write arbitrary content to WAL without a table, which can be used to advance the slot. A heartbeat does not help a consumer that is paused; it helps a consumer that is running but has nothing to read.

Dropping the slot is the correct action when no consumer will return. The logical decoding documentation is direct: if a slot is no longer required it should be dropped, and in extreme cases retained resources can cause the database to shut down to prevent transaction ID wraparound. A slot with no owner is a liability, not a safety net.

What to do when the alarm fires

Query pg_replication_slots first. Look at wal_status, inactive_since, and safe_wal_size. If wal_status is lost or invalidation_reason is non-NULL, the slot is already dead and the question is how to rebuild the consumer, not how to save the slot.

If the slot is still alive and the disk is filling, the decision rule is simple: will the consumer return within the window you sized the cap for? If yes, keep the slot and free space elsewhere — grow the volume, fix the archive backlog, move other data off the disk. If no, drop the slot and plan the re-snapshot. Do not gamble on the primary’s disk while waiting for a consumer that may not come back.

Do not delete files from pg_wal. Do not treat pg_resetwal as a recovery step; it is a last resort that risks data loss. If the disk is already full, add space before anything else, because a checkpoint may be required before dropping the slot frees anything.

FAQ

Does max_slot_wal_keep_size apply to physical replication slots too?

The documentation describes it as the maximum size of WAL files that replication slots are allowed to retain, and the pg_replication_slots view exposes wal_status and safe_wal_size for both physical and logical slots. The invalidation behavior is described in terms of a slot’s restart_lsn falling behind the current LSN, which applies to either type.

Why is safe_wal_size NULL on my cluster?

Two documented reasons: the slot is already lost, or max_slot_wal_keep_size is -1. If it is the latter, you have no countdown and the disk is your only limit.

Can I change the setting without restarting?

Yes. The documentation states it can only be set in postgresql.conf or on the server command line, which rules out session-level SET. But editing postgresql.conf and reloading the configuration applies the new value without a restart. Plan the change for a quiet moment, and plan what the new value means for any slot that is currently far behind.

Will the slot be invalidated the moment it crosses the limit?

No. The cap is applied at checkpoint time. The gap between crossing the limit and invalidation is bounded by your checkpoint interval, which is why alerting on safe_wal_size shrinking gives you more warning than alerting on disk percentage.

What does the consumer see after invalidation?

The slot is no longer usable. For a logical CDC connector, that typically means creating a new slot and taking a fresh initial snapshot, per Morling’s description of the Debezium case. The consumer cannot resume from its last confirmed position because the WAL it needed has been removed.

Sources

• PostgreSQL documentation, Replication configuration — max_slot_wal_keep_size, wal_keep_size, idle_replication_slot_timeout.
• PostgreSQL documentation, pg_replication_slots — wal_status, safe_wal_size, invalidation_reason, inactive_since.
• PostgreSQL documentation, Logical Decoding Concepts — slot persistence and resource retention.
• PostgreSQL documentation, WAL Configuration — max_wal_size as a soft limit, slot-driven accumulation.
• PostgreSQL documentation, System Administration Functions — pg_current_wal_lsn(), pg_wal_lsn_diff().
• Netdata, pg_wal directory full: causes and emergency recovery — checkpoint failure, recovery ordering, the prohibition on deleting WAL files.
• Gunnar Morling, Mastering Postgres Replication Slots — slot invalidation consequences, idle-database retention, pg_logical_emit_message() heartbeats.