apache/flink · error · IOException

Failed to deserialize Avro record.

Error message

Failed to deserialize Avro record.

What it means

AvroRowDataDeserializationSchema.deserialize wraps any failure of the nested Avro decoder or the GenericRecord->RowData runtime converter in an IOException. The root cause is in the suppressed exception: usually bytes that do not match the reader schema, or a value the converter cannot map to the declared logical type.

Source

Thrown at flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroRowDataDeserializationSchema.java:143

        this.typeInfo = typeInfo;
        this.runtimeConverter = runtimeConverter;
    }

    @Override
    public void open(InitializationContext context) throws Exception {
        this.nestedSchema.open(context);
    }

    @Override
    public RowData deserialize(@Nullable byte[] message) throws IOException {
        if (message == null) {
            return null;
        }
        try {
            GenericRecord deserialize = nestedSchema.deserialize(message);
            return (RowData) runtimeConverter.convert(deserialize);
        } catch (Exception e) {
            throw new IOException("Failed to deserialize Avro record.", e);
        }
    }

    @Override
    public boolean isEndOfStream(RowData nextElement) {
        return false;
    }

    @Override
    public TypeInformation<RowData> getProducedType() {
        return typeInfo;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) {
            return true;
        }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Inspect the cause chain (e.getCause()) to see whether it is a decode error (schema mismatch) or a converter error (value type), and align the consumer's Avro schema with the producer's writer schema.
2. Verify the incoming payload is Avro binary (not JSON/Avro JSON) and matches the schema registered/configured in the AvroRowDataDeserializationSchema.
3. If schemas legitimately evolved, configure the deserializer with the reader schema that is compatible with the writer schema used by the producer.
4. Add a dirty-record side output: catch IOException in a flatMap before the format and route bad bytes to a DLQ instead of failing the job.

Example fix

// before
RowData row = deserializationSchema.deserialize(bytes);

// after
RowData row;
try {
    row = deserializationSchema.deserialize(bytes);
} catch (IOException e) {
    log.warn("Bad Avro record: {}", e.getCause().getMessage());
    ctx.output(DLQ_TAG, bytes);
    return;
}

try {
    RowData row = deserializer.deserialize(bytes);
    collect(row);
} catch (IOException e) {
    // e.getCause() distinguishes decode failure vs converter failure
    ctx.output(deadLetterTag, bytes); // route raw bytes to DLQ
}

When it happens

Trigger: Calling deserialize() on bytes produced with a different/evolved Avro schema (field added/removed/type changed), corrupt or truncated Kafka payload, non-Avro bytes fed to the format, or an Avro logical type value the RowData converter does not recognize.

Common situations: Schema evolution without compatible reader/writer schemas in the Avro format DDL; producer writing JSON-encoded Avro while consumer expects binary; topic mix-up; toggling 'avro.timestamp.logical-type.mapping.legacy' between producer and consumer.

Related errors

• Unknown data format. Magic number does not match
• Could not find schema with id %s in registry
• Could not register schema in registry
• Failed to deserialize JSON '%s'.
• Failed to deserialize an element from the source. If you are

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/e56b301560c8a22a. Report an issue: GitHub.