apache/druid · error · ParseException

Failed to read Avro message with schema id

Error message

Failed to read Avro message with schema id[%s]

What it means

This ParseException is thrown by InlineSchemasAvroBytesDecoder.parse when the Avro binary payload cannot be decoded into a GenericRecord using the schema looked up from the inline schema map for the given schema id. The message bytes themselves are malformed relative to the schema, or the configured inline schema does not match the data.

Solutions

1. Verify the schema registered in inlineSchemas for the reported schema id matches the producer's schema (use the producer's .avsc)
2. Inspect raw message bytes to confirm they are valid Avro binary data
3. Ensure producer and consumer use compatible Avro writer/reader schema evolution (e.g., add fields with defaults)
4. Re-ingest the offending messages after fixing the schema configuration

Example fix

// before
"inlineSchemas": {"1": "{\"namespace\":\"old\",...}"}
// after
"inlineSchemas": {"1": "{\"namespace\":\"new\",\"fields\":[{\"name\":\"extraField\",\"type\":\"string\",\"default\":\"\"}, ...]}"}

// before ingestion, verify schema compatibility offline
Schema writer = new Schema.Parser().parse(new File("writer.avsc"));
Schema reader = new Schema.Parser().parse(configuredInlineSchema);
if (!writer.getFullName().equals(reader.getFullName()) || !reader.isCompatible(writer)) {
  throw new IllegalArgumentException("Inline schema does not match producer schema");
}

boolean isValidAvroPayload(ByteBuffer bytes) {
  return bytes != null && bytes.remaining() > 0;
}

try {
  GenericRecord record = decoder.parse(bytes);
} catch (ParseException e) {
  LOG.warn(e, "Dropping malformed Avro message");
  metrics.counter("avro-parse-failures").inc();
  // skip or route to DLQ
}

When it happens

Trigger: Calling parse(ByteBuffer bytes) where reader.read() fails: payload bytes corrupted/truncated, wrong schema registered for the schema id in inlineSchemas, or payload was not Avro binary-encoded.

Common situations: Kafka topics with multiple producers using different schemas under the same id; schema evolved but the inline schema map was not updated; non-Avro bytes sent to the topic by mistake.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Failed to decode Avro message for schema id
• Failed to read Avro message
• Failed to read Avro message
• Avro + JQ not supported
• Avro + nested tree extraction not supported

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f1234ac42c4be94b. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/InlineSchemasAvroBytesDecoder.java:122

    }

    byte version = bytes.get();
    if (version != V1) {
      throw new ParseException(null, "Found record of arbitrary version[%s]", version);
    }

    int schemaId = bytes.getInt();
    Schema schemaObj = schemaObjs.get(schemaId);
    if (schemaObj == null) {
      throw new ParseException(null, "Failed to find schema for id[%s]", schemaId);
    }

    DatumReader<GenericRecord> reader = new GenericDatumReader<>(schemaObj);
    try (ByteBufferInputStream inputStream = new ByteBufferInputStream(Collections.singletonList(bytes))) {
      return reader.read(null, DecoderFactory.get().binaryDecoder(inputStream, null));
    }
    catch (Exception e) {
      throw new ParseException(null, e, "Failed to read Avro message with schema id[%s]", schemaId);
    }
  }
}

View on GitHub (pinned to 9b90983fd2)