apache/flink · error · IllegalStateException

The "before" field of %s message is null, if you are using D

Error message

The "before" field of %s message is null, if you are using Debezium Postgres Connector, please check the Postgres table has been set REPLICA IDENTITY to FULL level.

What it means

IllegalStateException from DebeziumAvroDeserializationSchema: an UPDATE (or DELETE) message arrived whose 'before' field is null. Debezium's Postgres connector only populates 'before' when the table's REPLICA IDENTITY is FULL; otherwise UPDATE/DELETE rows carry no before-image and Flink cannot emit the required UPDATE_BEFORE/DELETE row, so processing fails.

Source

Thrown at flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/debezium/DebeziumAvroDeserializationSchema.java:146

    @Override
    public void deserialize(byte[] message, Collector<RowData> out) throws IOException {

        if (message == null || message.length == 0) {
            // skip tombstone messages
            return;
        }
        try {
            GenericRowData row = (GenericRowData) avroDeserializer.deserialize(message);

            GenericRowData before = (GenericRowData) row.getField(0);
            GenericRowData after = (GenericRowData) row.getField(1);
            String op = row.getField(2).toString();
            if (OP_CREATE.equals(op) || OP_READ.equals(op)) {
                after.setRowKind(RowKind.INSERT);
                out.collect(after);
            } else if (OP_UPDATE.equals(op)) {
                if (before == null) {
                    throw new IllegalStateException(
                            String.format(REPLICA_IDENTITY_EXCEPTION, "UPDATE"));
                }
                before.setRowKind(RowKind.UPDATE_BEFORE);
                after.setRowKind(RowKind.UPDATE_AFTER);
                out.collect(before);
                out.collect(after);
            } else if (OP_DELETE.equals(op)) {
                if (before == null) {
                    throw new IllegalStateException(
                            String.format(REPLICA_IDENTITY_EXCEPTION, "DELETE"));
                }
                before.setRowKind(RowKind.DELETE);
                out.collect(before);
            } else {
                throw new IOException(
                        format(
                                "Unknown \"op\" value \"%s\". The Debezium Avro message is '%s'",
                                op, new String(message)));

View on GitHub (pinned to 2f3c205e92)

Solutions

Run: ALTER TABLE <table> REPLICA IDENTITY FULL; on the source Postgres table, then reprocess affected messages.
New tables need the same ALTER after creation — automate it in your migration tooling.
Ensure the Debezium connector has REPLICA IDENTITY rights (owner or pg_role with replica).

Example fix

-- before (default)
-- table ships only PK in before-image; UPDATE with null before fails

-- after
ALTER TABLE orders REPLICA IDENTITY FULL;

Defensive patterns

Strategy: validation

Validate before calling

// before starting the Flink job, verify source table replica identity
// SELECT relreplident FROM pg_class WHERE relname = 'orders';
// 'd' = DEFAULT (unsafe), 'f' = FULL (required)
if (!"f".equals(queryPg("SELECT relreplident FROM pg_class WHERE relname='orders'"))) {
    throw new IllegalStateException(
        "orders must be ALTER TABLE orders REPLICA IDENTITY FULL before Debezium CDC");
}

Type guard

boolean replicaIdentityFull(String replIdent) { return "f".equals(replIdent); }

Try / catch

try {
    consumer.run();
} catch (IllegalStateException e) {
    if (e.getMessage().contains("REPLICA IDENTITY")) {
        // source-side fix required: ALTER TABLE <t> REPLICA IDENTITY FULL; then reprocess
        log.error("Postgres replica identity too weak for CDC; before-image missing", e);
    }
    throw e;
}

Prevention

Set REPLICA IDENTITY FULL on every CDC-enabled Postgres table at provisioning time (Terraform/Flyway step).
Add a pre-deploy SQL check on pg_class.relreplident for all captured tables.
Monitor for schema-drift: new tables added to the Debezium capture list need the same ALTER.

When it happens

Trigger: Consuming Debezium Postgres CDC (avro-confluent-registry + debezium-avro) where ALTER TABLE ... REPLICA IDENTITY has not been set to FULL; tables created after the connector started with default replica identity.

Common situations: Newly CDC-enabled Postgres tables default to DEFAULT replica identity (only primary key in before-image, null for non-PK or when key missing); DBAs skipping the replica identity step in deployment runbooks.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/736d5627e57a2884. Report an issue: GitHub.