apache/druid · error · ParseException

Record must have at least 5 bytes carrying version and schem

Error message

Record must have at least 5 bytes carrying version and schemaId

What it means

InlineSchemasAvroBytesDecoder expects Confluent wire format: 1 version byte + 4 schema-id bytes + Avro data, so every record must be at least 5 bytes. Shorter buffers cannot carry the version/schemaId header and throw a ParseException.

Source

Thrown at extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/InlineSchemasAvroBytesDecoder.java:103

    this.schemaObjs = schemaObjs;
    this.schemas = null;
  }

  @JsonProperty
  public Map<String, Map<String, Object>> getSchemas()
  {
    return schemas;
  }

  // It is assumed that record has following format.
  // byte 1 : version, static 0x1
  // byte 2-5 : int schemaId
  // remaining bytes would have avro data
  @Override
  public GenericRecord parse(ByteBuffer bytes)
  {
    if (bytes.remaining() < 5) {
      throw new ParseException(null, "Record must have at least 5 bytes carrying version and schemaId");
    }

    byte version = bytes.get();
    if (version != V1) {
      throw new ParseException(null, "Found record of arbitrary version[%s]", version);
    }

    int schemaId = bytes.getInt();
    Schema schemaObj = schemaObjs.get(schemaId);
    if (schemaObj == null) {
      throw new ParseException(null, "Failed to find schema for id[%s]", schemaId);
    }

    DatumReader<GenericRecord> reader = new GenericDatumReader<>(schemaObj);
    try (ByteBufferInputStream inputStream = new ByteBufferInputStream(Collections.singletonList(bytes))) {
      return reader.read(null, DecoderFactory.get().binaryDecoder(inputStream, null));
    }
    catch (Exception e) {

View on GitHub (pinned to 9b90983fd2)

Solutions

Ensure producers use Confluent's serializer (KafkaAvroSerializer) so every message carries the 5-byte header; configure value.subject.name.strategy consistently.
Filter out empty/tombstone records before ingestion (e.g. a transform/filter in the ingestion spec or a compacted-topic-aware consumer).
Verify messages aren't truncated: check broker/producer max message sizes and any intermediary that splits payloads.
If the topic mixes formats, separate non-Avro records onto a different topic or use a decoder that matches the actual wire format.

Example fix

// before
byte[] payload = new byte[0];
producer.send(new ProducerRecord<>(topic, payload)); // tombstone-style
// after
if (avroRecord != null) {
  byte[] payload = serializeWithConfluentWireFormat(avroRecord); // magic+schemaId+data
  producer.send(new ProducerRecord<>(topic, payload));
}

Defensive patterns

Strategy: validation

Validate before calling

// Reject malformed records before they reach ingestion
if (message == null || message.length < 5) {
  throw new IllegalArgumentException("Kafka record must carry >=5 bytes: version byte + schemaId + avro data");
}

Type guard

boolean isWireFramed(byte[] msg) {
  return msg != null && msg.length >= 5 && msg[0] == 0;
}

Try / catch

try {
  GenericRecord r = decoder.parse(ByteBuffer.wrap(msg));
} catch (ParseException e) {
  if (e.getMessage().contains("at least 5 bytes")) {
    deadLetter(msg, "short/no-header record"); // skip tombstones/control messages
  }
}

Prevention

Never send null/empty or tombstone records to Avro-ingested topics; use a separate topic for tombstones.
Produce with KafkaAvroSerializer so headers are always present.
Check broker retention/segment settings that could truncate messages.
Add a consumer-side length guard before decoding.

When it happens

Trigger: parse(ByteBuffer) is handed a record with fewer than 5 remaining bytes — an empty or nearly-empty message, a message produced without the Confluent wire-format header, or a truncated message at the tail of a batch/compacted topic.

Common situations: Tombstone/empty Kafka messages or control messages sent to an Avro topic; a producer writing plain Avro binary (no schema id header) consumed with multiple_schemas decoding; truncation caused by an upstream chunking bug.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f2e6f0095c1366a4. Report an issue: GitHub.