{"record":{"id":"b9f1571db34cf95a","repo":"provectus/kafka-ui","slug":"data-doesn-t-contain-magic-byte-and-schema-id-pref","errorCode":null,"errorMessage":"Data doesn't contain magic byte and schema id prefix, so it can't be deserialized with %s serde","messagePattern":"Data doesn't contain magic byte and schema id prefix, so it can't be deserialized with (.+?) serde","errorType":"validation","errorClass":"ValidationException","httpStatus":400,"severity":"error","filePath":"kafka-ui-api/src/main/java/com/provectus/kafka/ui/serdes/builtin/sr/SchemaRegistrySerde.java","lineNumber":308,"sourceCode":"              \"type\", format.name()\n          )\n      );\n    };\n  }\n\n  private SchemaType getMessageFormatBySchemaId(int schemaId) {\n    return getSchemaById(schemaId)\n        .map(ParsedSchema::schemaType)\n        .flatMap(SchemaType::fromString)\n        .orElseThrow(() -> new ValidationException(String.format(\"Schema for id '%d' not found \", schemaId)));\n  }\n\n  private int extractSchemaIdFromMsg(byte[] data) {\n    ByteBuffer buffer = ByteBuffer.wrap(data);\n    if (buffer.remaining() >= SR_PAYLOAD_PREFIX_LENGTH && buffer.get() == SR_PAYLOAD_MAGIC_BYTE) {\n      return buffer.getInt();\n    }\n    throw new ValidationException(\n        String.format(\n            \"Data doesn't contain magic byte and schema id prefix, so it can't be deserialized with %s serde\",\n            name())\n    );\n  }\n}\n","sourceCodeStart":290,"sourceCodeEnd":315,"githubUrl":"https://github.com/provectus/kafka-ui/blob/83b5a60cc08501b570a0c4d0b4cdfceb1b88d6b7/kafka-ui-api/src/main/java/com/provectus/kafka/ui/serdes/builtin/sr/SchemaRegistrySerde.java#L290-L315","documentation":"The SchemaRegistrySerde expects messages in the Confluent Schema Registry wire format: a 0x0 magic byte followed by a 4-byte schema id prefix. This error is thrown by extractSchemaIdFromMsg when the byte array is shorter than 5 bytes or its first byte is not the magic byte, so the payload cannot be mapped to a registered schema for deserialization.","triggerScenarios":"Calling the SchemaRegistrySerde deserializer (via schemaId) on message bytes that were not produced with the Confluent wire format - e.g. raw JSON/Avro bytes, plain string payloads, or records from a custom serializer that skips the magic byte/schema id prefix.","commonSituations":"A topic holds plain-text or non-registry-serialized messages (produced with StringSerializer or raw clients) but is consumed in kafka-ui with the Schema Registry serde; a producer changed to write raw Avro without the Confluent prefix; reading migrated/compacted data where the prefix was stripped.","solutions":["Select a different serde in kafka-ui (String, Raw, or Json) for topics whose messages lack the Schema Registry prefix","Fix the producer to serialize with Confluent's KafkaAvroSerializer/KafkaJsonSerializer so the magic byte and schema id are prepended","Check the first byte of your messages: it must be 0x0 followed by a 4-byte big-endian schema id; re-produce affected records if malformed"],"exampleFix":"// before (producer writes raw Avro, no prefix)\nbyte[] value = avroEncoder.encode(record);\nproducer.send(new ProducerRecord<>(topic, value));\n// after (use Confluent serializer which adds magic byte + schema id)\nprops.put(\"value.serializer\", \"io.confluent.kafka.serializers.KafkaAvroSerializer\");\nprops.put(\"schema.registry.url\", \"http://schema-registry:8081\");","handlingStrategy":"validation","validationCode":"boolean hasSchemaRegistryPrefix(byte[] data) {\n  ByteBuffer buf = ByteBuffer.wrap(data);\n  return buf.remaining() >= 5 && buf.get() == 0x0;\n}","typeGuard":"boolean isSrEncoded(byte[] data) {\n  return data != null && data.length >= 5 && data[0] == 0x0;\n}","tryCatchPattern":"try {\n  byte[] decoded = serde.deserialize(topic, raw);\n} catch (ValidationException e) {\n  log.warn(\"Payload lacks SR prefix, falling back to raw\");\n  decoded = raw;\n}","preventionTips":["Produce with Confluent serializers (KafkaAvroSerializer/KafkaJsonSerializer/KafkaProtobufSerializer) so the wire prefix is always present","Before choosing the Schema Registry serde in kafka-ui, inspect a message with the Raw/String serde to confirm the 0x0 prefix","Never strip or transform the first 5 bytes of registry-encoded records in bridge/proxy tools"],"tags":["kafka","schema-registry","deserialization","avro"],"backgroundTag":"invalid-argument-format","analyzedSha":"83b5a60cc08501b570a0c4d0b4cdfceb1b88d6b7","analyzedAt":"2026-09-08T04:35:39.002Z","contentChangedAt":"2026-09-08T04:35:39.002Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}