provectus/kafka-ui · error · ValidationException

Data doesn't contain magic byte and schema id prefix, so it…

Error message

Data doesn't contain magic byte and schema id prefix, so it can't be deserialized with %s serde

What it means

The SchemaRegistrySerde expects messages in the Confluent Schema Registry wire format: a 0x0 magic byte followed by a 4-byte schema id prefix. This error is thrown by extractSchemaIdFromMsg when the byte array is shorter than 5 bytes or its first byte is not the magic byte, so the payload cannot be mapped to a registered schema for deserialization.

Solutions

Select a different serde in kafka-ui (String, Raw, or Json) for topics whose messages lack the Schema Registry prefix
Fix the producer to serialize with Confluent's KafkaAvroSerializer/KafkaJsonSerializer so the magic byte and schema id are prepended
Check the first byte of your messages: it must be 0x0 followed by a 4-byte big-endian schema id; re-produce affected records if malformed

Example fix

// before (producer writes raw Avro, no prefix)
byte[] value = avroEncoder.encode(record);
producer.send(new ProducerRecord<>(topic, value));
// after (use Confluent serializer which adds magic byte + schema id)
props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
props.put("schema.registry.url", "http://schema-registry:8081");

Defensive patterns

Strategy: validation

Validate before calling

boolean hasSchemaRegistryPrefix(byte[] data) {
  ByteBuffer buf = ByteBuffer.wrap(data);
  return buf.remaining() >= 5 && buf.get() == 0x0;
}

Type guard

boolean isSrEncoded(byte[] data) {
  return data != null && data.length >= 5 && data[0] == 0x0;
}

Try / catch

try {
  byte[] decoded = serde.deserialize(topic, raw);
} catch (ValidationException e) {
  log.warn("Payload lacks SR prefix, falling back to raw");
  decoded = raw;
}

Prevention

Produce with Confluent serializers (KafkaAvroSerializer/KafkaJsonSerializer/KafkaProtobufSerializer) so the wire prefix is always present
Before choosing the Schema Registry serde in kafka-ui, inspect a message with the Raw/String serde to confirm the 0x0 prefix
Never strip or transform the first 5 bytes of registry-encoded records in bridge/proxy tools

When it happens

Trigger: Calling the SchemaRegistrySerde deserializer (via schemaId) on message bytes that were not produced with the Confluent wire format - e.g. raw JSON/Avro bytes, plain string payloads, or records from a custom serializer that skips the magic byte/schema id prefix.

Common situations: A topic holds plain-text or non-registry-serialized messages (produced with StringSerializer or raw clients) but is consumed in kafka-ui with the Schema Registry serde; a producer changed to write raw Avro without the Confluent prefix; reading migrated/compacted data where the prefix was stripped.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of provectus/kafka-ui@83b5a60cc0 (2026-09-08). Data as JSON: /api/errors/b9f1571db34cf95a. Report an issue: GitHub.

Appendix: source

Thrown at kafka-ui-api/src/main/java/com/provectus/kafka/ui/serdes/builtin/sr/SchemaRegistrySerde.java:308

              "type", format.name()
          )
      );
    };
  }

  private SchemaType getMessageFormatBySchemaId(int schemaId) {
    return getSchemaById(schemaId)
        .map(ParsedSchema::schemaType)
        .flatMap(SchemaType::fromString)
        .orElseThrow(() -> new ValidationException(String.format("Schema for id '%d' not found ", schemaId)));
  }

  private int extractSchemaIdFromMsg(byte[] data) {
    ByteBuffer buffer = ByteBuffer.wrap(data);
    if (buffer.remaining() >= SR_PAYLOAD_PREFIX_LENGTH && buffer.get() == SR_PAYLOAD_MAGIC_BYTE) {
      return buffer.getInt();
    }
    throw new ValidationException(
        String.format(
            "Data doesn't contain magic byte and schema id prefix, so it can't be deserialized with %s serde",
            name())
    );
  }
}

View on GitHub (pinned to 83b5a60cc0)