risingwavelabs/risingwave · error · WireFormatError::NoMagic

failed to match the magic byte 0

Error message

failed to match the magic byte 0

What it means

WireFormatError::NoMagic is thrown when decoding a Confluent wire-format message payload: the first (magic) byte of the message is not 0, which is the only version the decoder supports. This means the payload does not follow the Confluent Schema Registry envelope (magic byte + 4-byte schema ID + payload).

Source

Thrown at src/connector/src/schema/schema_registry/util.rs:48

        match ele.parse::<Url>() {
            Ok(url) => urls.push(url),
            Err(e) => errs.push(e),
        }
    }
    if urls.is_empty() {
        bail_invalid_option_error!("no valid url provided, errs: {errs:?}");
    }
    tracing::debug!(
        "schema registry client will use url {:?} to connect, the rest failed because: {:?}",
        urls,
        errs
    );
    Ok(urls)
}

#[derive(Debug, thiserror::Error)]
pub enum WireFormatError {
    #[error("failed to match the magic byte 0")]
    NoMagic,
    #[error("failed to read the 4-byte schema ID")]
    NoSchemaId,
    #[error("failed to parse message indexes")]
    ParseMessageIndexes,
}

/// Returns `(schema_id, payload)`
///
/// Refer to [Confluent schema registry wire format](https://docs.confluent.io/platform/7.6/schema-registry/fundamentals/serdes-develop/index.html#wire-format)
///
/// | Bytes | Area        | Description                                                                                        |
/// |-------|-------------|----------------------------------------------------------------------------------------------------|
/// | 0     | Magic Byte  | Confluent serialization format version number; currently always `0`.                               |
/// | 1-4   | Schema ID   | 4-byte schema ID as returned by Schema Registry.                                                   |
/// | 5-... | Data        | Serialized data for the specified schema format (for example, binary encoding for Avro or Protobuf.|
pub(crate) fn extract_schema_id(payload: &[u8]) -> Result<(i32, &[u8]), WireFormatError> {
    use byteorder::{BigEndian, ReadBytesExt as _};

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the producer uses a Confluent-compatible serializer (KafkaAvroSerializer / KafkaJsonSchemaSerializer / Protobuf with registry) and points at the same registry.
2. Inspect a raw message's first byte (hex) to confirm the envelope; if absent, re-produce data with the registry serializer.
3. If tombstones are expected, filter out null/empty payloads before decoding.
4. Check that no producer upgrades changed the wire format for this topic.

Example fix

// before: decoding every message
let (id, payload) = parse_wire_format(bytes)?;
// after: skip non-envelope payloads
if bytes.first() == Some(&0) {
    let (id, payload) = parse_wire_format(bytes)?;
}

fn has_confluent_envelope(bytes: &[u8]) -> bool {
    bytes.first() == Some(&0)
}

fn is_wire_format_v0(b: &[u8]) -> bool { b.first() == Some(&0) && b.len() >= 5 }

match parse_wire_format(bytes) {
    Ok((id, payload)) => decode(id, payload),
    Err(WireFormatError::NoMagic) => {
        tracing::warn!("non-registry payload; skipping");
        skip();
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Parsing a Kafka message value that does not begin with byte 0 — e.g. messages produced without the schema registry serializer, plain JSON/Avro without the envelope, or tombstones/empty messages being fed to the decoder.

Common situations: Producing to the topic with a non-registry serializer (e.g. plain JsonProducer), messages from a producer using a newer wire format, or misconfigured topic where mixed message formats exist.

Related errors

• failed to read the 4-byte schema ID
• Must specify '{}' or '{}'
• failed to parse message indexes
• confluent schema registry error {error_code}: {message}
• request error

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/181c531e124a3d66. Report an issue: GitHub.