apache/druid · error · ParseException

Found record of arbitrary version[%s]

Error message

Found record of arbitrary version[%s]

What it means

The first byte of each record is a format version byte; currently only version V1 (the Confluent-compatible wire format) is understood. A record whose version byte differs is rejected with a ParseException naming the offending version.

Source

Thrown at extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/InlineSchemasAvroBytesDecoder.java:108

  public Map<String, Map<String, Object>> getSchemas()
  {
    return schemas;
  }

  // It is assumed that record has following format.
  // byte 1 : version, static 0x1
  // byte 2-5 : int schemaId
  // remaining bytes would have avro data
  @Override
  public GenericRecord parse(ByteBuffer bytes)
  {
    if (bytes.remaining() < 5) {
      throw new ParseException(null, "Record must have at least 5 bytes carrying version and schemaId");
    }

    byte version = bytes.get();
    if (version != V1) {
      throw new ParseException(null, "Found record of arbitrary version[%s]", version);
    }

    int schemaId = bytes.getInt();
    Schema schemaObj = schemaObjs.get(schemaId);
    if (schemaObj == null) {
      throw new ParseException(null, "Failed to find schema for id[%s]", schemaId);
    }

    DatumReader<GenericRecord> reader = new GenericDatumReader<>(schemaObj);
    try (ByteBufferInputStream inputStream = new ByteBufferInputStream(Collections.singletonList(bytes))) {
      return reader.read(null, DecoderFactory.get().binaryDecoder(inputStream, null));
    }
    catch (Exception e) {
      throw new ParseException(null, e, "Failed to read Avro message with schema id[%s]", schemaId);
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Confirm the producer's wire-format magic byte is 0 (Confluent v1); reconfigure non-standard serializers to the Confluent wire format.
  2. Upgrade Druid/avro-extensions to a version that supports the record's wire-format version if a newer format is in use.
  3. Audit the topic for mixed serialization; split foreign-format records to another topic.
  4. Validate a failing message offline: dump the first bytes and check the expected 0 + 4-byte schema-id layout.

Example fix

// before (raw Avro binary, no header)
DatumWriter w = new GenericDatumWriter(schema); w.write(record, enc); // first byte is Avro data
// after
out.write(0); // Confluent magic/version byte V1
out.writeIntBE(schemaId);
DatumWriter w = new GenericDatumWriter(schema); w.write(record, enc);
Defensive patterns

Strategy: validation

Validate before calling

// Check the version byte before decoding
byte version = msg[0];
if (version != 0 /* V1 */) {
  throw new IllegalArgumentException("Unsupported Avro wire-format version " + version + "; expected Confluent V1 (0)");
}

Type guard

boolean isV1WireFormat(byte[] msg) {
  return msg != null && msg.length > 0 && msg[0] == 0;
}

Try / catch

try {
  GenericRecord r = decoder.parse(ByteBuffer.wrap(msg));
} catch (ParseException e) {
  if (String.valueOf(e.getMessage()).contains("arbitrary version")) {
    deadLetter(msg, "unsupported wire-format version");
  }
}

Prevention

When it happens

Trigger: parse(ByteBuffer) reads a record whose first byte != V1 — messages written by a newer Confluent/serializer wire-format version, a completely different serialization whose first byte collides differently (e.g. raw Avro binary data starting with a non-magic byte), or byte-offset corruption shifting the header.

Common situations: Upgrading serializers (schema registry clients) to a wire format the Druid decoder predates; mixed-format topics where some messages aren't Confluent-framed; misaligned reads after earlier parse bugs consumed wrong byte counts.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4b083e18b34aaac7. Report an issue: GitHub.