databendlabs/databend · error · InvalidData

custom-encoded value too short to contain 4-byte header

Error message

custom-encoded value too short to contain 4-byte header

What it means

decode_value in the meta kv pb_api implements a custom value encoding: a 1-byte MAGIC followed by a 4-byte header of flags. If a buffer starts with the MAGIC byte (so it claims to be custom-encoded) but is shorter than the 4-byte header, it is corrupt and decode_value returns io::ErrorKind::InvalidData with this message.

Solutions

1. Treat the value as corrupt: re-write the key with a fresh encode_value and re-run the operation.
2. Verify the meta store's backing storage for truncation/corruption; restore from a snapshot/backup if records are damaged.
3. If the value legitimately starts with the magic byte, ensure it was produced by encode_value so the full header is present.

fn looks_like_custom_value(buf: &[u8]) -> bool {
    buf.first() == Some(&MAGIC) && buf.len() >= HEADER_LEN
}

let value = match decode_value(&raw) {
    Ok(v) => v,
    Err(e) if e.kind() == io::ErrorKind::InvalidData => {
        // treat as corrupt; rebuild the record from source of truth
        reencode_and_store(key)?;
        Cow::Owned(default_value())
    }
    Err(e) => return Err(e.into()),
};

When it happens

Trigger: Calling decode_value on a buffer whose first byte equals MAGIC but whose length is 1-3 bytes — i.e. a truncated or corrupted meta storage value, or a value whose payload coincidentally begins with the MAGIC byte and got truncated.

Common situations: Corrupted or truncated underlying KV storage records (e.g. partial write, disk issue); hand-crafted test bytes starting with the magic byte; restore/copy of meta data that was cut short.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• unknown flags in custom-encoded value header
• Invalid version
• Access denied: is outside allowed directories
• {e}
• e

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/47695a5bc8e670bb. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/api/src/kv/pb_api/compress.rs:122

/// Global encoder instance, used by [`codec::encode_pb`].
///
/// Compression is disabled by default. Call [`GLOBAL_ENCODER.set_compress(true)`]
/// at startup to enable it.
pub static GLOBAL_ENCODER: Encoder = Encoder::new(false);

/// Decode a value produced by [`Encoder::encode_value`] or a legacy raw protobuf value.
///
/// Returns `Cow::Borrowed` for legacy/uncompressed data (zero copy),
/// and `Cow::Owned` for decompressed data.
/// Returns an error if the header has unknown flag bits set (forward-compat guard).
pub fn decode_value(buf: &[u8]) -> Result<Cow<'_, [u8]>, io::Error> {
    if buf.is_empty() || buf[0] != MAGIC {
        return Ok(Cow::Borrowed(buf));
    }

    if buf.len() < HEADER_LEN {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            "custom-encoded value too short to contain 4-byte header",
        ));
    }

    let flag1 = buf[1];
    let flag2 = buf[2];
    let flag3 = buf[3];

    let unknown = (flag1 & !FLAG_ZSTD) | flag2 | flag3;
    if unknown != 0 {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            format!(
                "unknown flags in custom-encoded value header: [{:#04x}, {:#04x}, {:#04x}]",
                flag1, flag2, flag3
            ),
        ));

View on GitHub (pinned to 288d84d76e)