apache/druid · error · ParseException

Failed to decode Avro message for schema id

Error message

Failed to decode Avro message for schema id[%s]

What it means

Thrown when the Avro schema was successfully fetched from the registry but the binary message still fails to decode via GenericDatumReader.read(). The payload does not conform to the schema registered under its id.

Solutions

1. Compare the writer schema of the failing messages with the registry schema for that id and fix any incompatible change (use full/backward compatibility mode on the registry subject)
2. Check for message truncation: increase Kafka max.message.bytes / fetch sizes if records are large
3. Inspect a raw failing message with a standalone Avro decoder to isolate the byte offset
4. Reproduce with the same GenericDatumReader to validate the fix offline

Example fix

null

// decode offline to reproduce and localize the failure
DatumReader<GenericRecord> r = new GenericDatumReader<>(registrySchema);
Decoder d = DecoderFactory.get().binaryDecoder(rawBytes, null);
r.read(null, d); // throws here if bytes don't match schema

try {
  GenericRecord record = decoder.parse(bytes);
} catch (ParseException e) {
  if (e.getMessage().startsWith("Failed to decode Avro message")) {
    LOG.warn(e, "Message does not conform to registered schema; sending to DLQ");
  }
}

When it happens

Trigger: parse(ByteBuffer bytes) calls reader.read() with the registry schema and it throws — message bytes corrupted, written with a different/incompatible schema version than the one registered under that id, or truncation (e.g., oversized record cut off).

Common situations: Schema registry entries overwritten with non-backwards-compatible schemas; message truncation by broker/proxy (fetch.max.bytes limits); producer bugs writing malformed records; mixing Avro serialization configs (e.g., different normalize settings).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Failed to authenticate to schema registry for Avro schema id
• Failed to decode avro message, not enough bytes to decode
• Failed to fetch Avro schema id
• Failed to fetch Avro schema id
• Failed to find schema for id

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/867ee5d6e303ce4a. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/SchemaRegistryBasedAvroBytesDecoder.java:179

      // For all other errors, just include the code and message received from the library.
      throw new ParseException(
          null,
          ex2,
          "Failed to fetch Avro schema id[%s] from registry. Error code[%s] and message[%s].",
          id,
          ex2.getErrorCode(),
          ex2.getMessage()
      );
    }
    if (schema == null) {
      throw new ParseException(null, "No Avro schema id[%s] in registry", id);
    }
    DatumReader<GenericRecord> reader = new GenericDatumReader<>(schema);
    try {
      return reader.read(null, DecoderFactory.get().binaryDecoder(bytes.array(), offset, length, null));
    }
    catch (Exception e) {
      throw new ParseException(null, e, "Failed to decode Avro message for schema id[%s]", id);
    }
  }

  @Override
  public boolean equals(Object o)
  {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }
    SchemaRegistryBasedAvroBytesDecoder that = (SchemaRegistryBasedAvroBytesDecoder) o;
    return capacity == that.capacity
           && Objects.equals(url, that.url)
           && Objects.equals(urls, that.urls)
           && Objects.equals(config, that.config)
           && Objects.equals(headers, that.headers);

View on GitHub (pinned to 9b90983fd2)