apache/druid · error · ParseException

Failed to decode avro message, not enough bytes to decode (%

Error message

Failed to decode avro message, not enough bytes to decode (%s)

What it means

This ParseException is thrown when the ByteBuffer received by SchemaRegistryBasedAvroBytesDecoder.parse is too short to contain the Confluent wire-format header: a 1-byte magic byte plus a 4-byte schema id. length = limit - 1 - 4 being negative means fewer than 5 bytes total.

Source

Thrown at extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/SchemaRegistryBasedAvroBytesDecoder.java:130

  //For UT only
  @VisibleForTesting
  SchemaRegistryBasedAvroBytesDecoder(SchemaRegistryClient registry)
  {
    this.url = null;
    this.capacity = Integer.MAX_VALUE;
    this.urls = null;
    this.config = null;
    this.headers = null;
    this.registry = registry;
    this.jsonMapper = new ObjectMapper();
  }

  @Override
  public GenericRecord parse(ByteBuffer bytes)
  {
    int length = bytes.limit() - 1 - 4;
    if (length < 0) {
      throw new ParseException(null, "Failed to decode avro message, not enough bytes to decode (%s)", bytes.limit());
    }

    bytes.get(); // ignore first \0 byte
    int id = bytes.getInt(); // extract schema registry id
    int offset = bytes.position() + bytes.arrayOffset();
    Schema schema;

    try {
      ParsedSchema parsedSchema = registry.getSchemaById(id);
      schema = parsedSchema instanceof AvroSchema ? ((AvroSchema) parsedSchema).rawSchema() : null;
    }
    catch (IOException ex1) {
      throw new ParseException(
          null,
          ex1,
          "Failed to fetch Avro schema id[%s] from registry. Check if the schema exists in the registry. Otherwise it"
          + " could mean that there is malformed data in the stream or data that doesn't conform to the schema"
          + " specified.",

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Ensure the producer uses Confluent KafkaAvroSerializer so messages carry the magic byte and schema id
  2. Filter out empty or tombstone messages before ingestion
  3. If messages are raw Avro without the registry envelope, switch to a decoder that matches the actual format (e.g., InlineSchemasAvroBytesDecoder)
  4. Check byte order/serialization on the producer side for the id field

Example fix

// before: raw avro bytes without wire format sent to registry-based decoder
// after: producer config
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
props.put("schema.registry.url", "http://schema-registry:8081");
Defensive patterns

Strategy: validation

Validate before calling

// verify payload has Confluent wire format before decoding
boolean hasConfluentWireFormat(ByteBuffer buf) {
  return buf.remaining() >= 5 && buf.get(buf.position()) == 0;
}

Try / catch

try {
  GenericRecord record = decoder.parse(bytes);
} catch (ParseException e) {
  if (e.getMessage().contains("not enough bytes")) {
    LOG.warn("Message missing Confluent wire-format header; producer misconfigured?");
  }
}

Prevention

When it happens

Trigger: parse(ByteBuffer bytes) called with a buffer whose limit is less than 5 bytes — e.g., empty messages, messages missing the Confluent schema-registry magic byte and id prefix, or raw Avro binary without the wire-format envelope.

Common situations: Producer not using KafkaAvroSerializer (writes raw Avro without the 5-byte header); tombstone/empty records in compacted topics; a connector delivering plain Avro payloads to a decoder configured for Confluent wire format.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8c65b6f8d29ccb9a. Report an issue: GitHub.