apache/seatunnel · error · ConnectException

The offset to start reading from has been removed from the d

Error message

The offset to start reading from has been removed from the database write-ahead log. Create a new snapshot and consider setting of PostgreSQL parameter wal_keep_segments = 0.

What it means

PostgreSQL has already recycled/removed the WAL segment containing the slot's confirmed flush LSN, so the replication stream cannot be rewound to the stored offset. Debezium (bundled in connector-cdc-opengauss) detects the server's 'requested WAL segment ... has already been removed' reply during startPgReplicationStream and converts it into a ConnectException, because continuing would silently skip data. The only recovery is a new snapshot; wal_keep_segments/wal_keep_size only delays, not prevents, this.

Source

Thrown at seatunnel-connectors-v2/connector-cdc/connector-cdc-opengauss/src/main/java/io/debezium/connector/postgresql/connection/PostgresReplicationConnection.java:559

            }
        } catch (PSQLException e) {
            if (e.getMessage().matches("(?s)ERROR: option .* is unknown.*")) {
                // It is possible we are connecting to an old wal2json plug-in
                LOGGER.warn(
                        "Could not register for streaming with metadata in messages, falling back to messages without metadata");

                // re-init the slot after a failed start of slot, as this
                // may have closed the slot
                if (useTemporarySlot()) {
                    initReplicationSlot();
                }

                s = startPgReplicationStream(startLsn, messageDecoder::optionsWithoutMetadata);
                messageDecoder.setContainsMetadata(false);
            } else if (e.getMessage()
                    .matches("(?s)ERROR: requested WAL segment .* has already been removed.*")) {
                LOGGER.error("Cannot rewind to last processed WAL position", e);
                throw new ConnectException(
                        "The offset to start reading from has been removed from the database write-ahead log. Create a new snapshot and consider setting of PostgreSQL parameter wal_keep_segments = 0.");
            } else {
                throw e;
            }
        }

        final PGReplicationStream stream = s;

        return new ReplicationStream() {

            private static final int CHECK_WARNINGS_AFTER_COUNT = 100;
            private int warningCheckCounter = CHECK_WARNINGS_AFTER_COUNT;
            private ExecutorService keepAliveExecutor = null;
            private AtomicBoolean keepAliveRunning;
            private final Metronome metronome =
                    Metronome.sleeper(statusUpdateInterval, Clock.SYSTEM);

            // make sure this is volatile since multiple threads may be interested in this value

View on GitHub (pinned to cf67b549a7)

Solutions

1. Take a new snapshot (reset offsets/delete the connector state) so streaming restarts from a current LSN
2. Increase PostgreSQL wal_keep_size (or legacy wal_keep_segments) and/or max_slot_wal_keep_size so required segments survive outages
3. Reduce connector downtime: monitor replication slot lag (pg_replication_slots) and alert before WAL is recycled
4. If archive_mode is on, restore/purge policies can be tuned; ensure segments referenced by the slot are not manually deleted

Example fix

-- before (postgresql.conf)
wal_keep_size = 0
-- after
wal_keep_size = 512MB
max_slot_wal_keep_size = 10GB

SELECT slot_name, active, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes, safe_wal_size FROM pg_replication_slots WHERE slot_name = 'debezium';
-- also check: SHOW wal_keep_size; and that required segments exist:
SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) < 2147483648 AS wal_within_keep;

try {
    stream = connection.createReplicationStream();
} catch (ConnectException e) {
    if (e.getMessage().contains("removed from the database write-ahead log")) {
        // WAL segment gone: cannot resume — trigger full re-snapshot / reset offsets
        planner.recoverWithSnapshot();
    } else {
        throw e; // transient: allow retry
    }
}

When it happens

Trigger: createReplicationStream -> startStreaming calls startPgReplicationStream(startLsn,...) and the server responds 'ERROR: requested WAL segment ... has already been removed' because the connector was down (or the slot lagged) long enough for PostgreSQL to recycle the needed WAL segment.

Common situations: Connector stopped for hours/days while the database kept writing; too-small wal_keep_size/replication slot lag; WAL disk pressure forcing recycling; restart after a long paused pipeline; slot unused while archives were purged.

Related errors

• Interrupted while waiting for valid replication slot info
• Unable to obtain valid replication slot. Make sure there are
• Neither confirmed_flush_lsn nor restart_lsn could be found
• restart_lsn could be found
• Invalid LSN returned from database

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/8d70f34ae749359c. Report an issue: GitHub.