apache/druid · error · ParseException

Failed to read Avro message

Error message

Failed to read Avro message

What it means

InlineSchemaAvroBytesDecoder embeds the Avro schema in each message (Confluent wire format) and then reads the record. Any exception during deserialization — corrupt bytes, wrong schema framing, truncation, or reader/writer schema mismatch — is wrapped in a ParseException with the message "Failed to read Avro message".

Source

Thrown at extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/InlineSchemaAvroBytesDecoder.java:90

    this.schemaObj = schemaObj;
    this.reader = new GenericDatumReader<>(schemaObj);
    this.schema = null;
  }

  @JsonProperty
  public Map<String, Object> getSchema()
  {
    return schema;
  }

  @Override
  public GenericRecord parse(ByteBuffer bytes)
  {
    try (ByteBufferInputStream inputStream = new ByteBufferInputStream(Collections.singletonList(bytes))) {
      return reader.read(null, DecoderFactory.get().binaryDecoder(inputStream, null));
    }
    catch (Exception e) {
      throw new ParseException(null, e, "Failed to read Avro message");
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify the topic truly contains Confluent wire-format Avro (byte 0 = 0, then 4-byte schema id) and that the inline_schema decoder matches your producer's serializer.
2. Check schema registry connectivity and that the schema id in the message resolves; fix URLs/credentials if the registry is unreachable.
3. Inspect and fix the producer (serializer config) or filter/requeue bad records; enable per-datasource error toleration (maxParseExceptions) to skip poison messages.
4. Validate one problematic message offline (deserialize with avro-tools or a small Java test) to pinpoint schema vs. payload corruption.

Example fix

// before (produce raw JSON to an Avro topic)
producer.send(record.value().toString());
// after
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.write(0); out.write(confluentSchemaIdBytes); out.write(binaryEncoder Avro datum);
producer.send(out.toByteArray());

// Offline pre-check of a message's framing
ByteBuffer bytes = ...;
if (bytes.remaining() < 5 || bytes.get(0) != 0) {
  throw new IllegalStateException("Message is not Confluent wire-format Avro; decoder would fail");
}

try {
  GenericRecord record = decoder.parse(bytes);
} catch (ParseException e) {
  log.error("Bad Avro message (cause: %s) — quarantine offset %d", e.getCause(), offset);
  // skip record / send to dead-letter topic
}

When it happens

Trigger: parse(ByteBuffer) receives bytes whose schema/record framing doesn't match: the embedded schema id can't be fetched from the schema registry, the payload is truncated/garbled (e.g. non-Avro messages on the topic), or a compatibility mismatch makes GenericDatumReader.read fail.

Common situations: Producing messages with a different serialization (e.g. raw JSON or protobuf) on a topic configured as Avro; schema registry returns an incompatible or missing schema; network failure reaching the registry mid-read; Kafka retention of corrupt/truncated messages.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Record must have at least 5 bytes carrying version and schem
• Cannot deserialize type[%s] to an RoaringBitmap64Counter:
• Avro + JQ not supported
• Avro + nested tree extraction not supported
• avroBytesDecoder is required to decode Avro records

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/806ad876466c1a1c. Report an issue: GitHub.