databendlabs/databend · error · InvalidData

unknown flags in custom-encoded value header

Error message

unknown flags in custom-encoded value header: [{:#04x}, {:#04x}, {:#04x}]

What it means

The custom value header reserves flag1 bit FLAG_ZSTD, while flag2 and flag3 are entirely unused. decode_value ORs together the unknown bits of all three flag bytes and rejects the value with InvalidData when any is set, so values encoded by a future/unknown encoder version are refused instead of being misinterpreted.

Solutions

1. Identify the writer that set the unknown flags (likely a newer version) and upgrade the reader's Databend version to match.
2. Re-encode the value with the current encode_value so only supported flags are set.
3. Check the stored bytes for corruption (compare against a backup/snapshot) if no version change occurred.

fn header_flags_known(buf: &[u8]) -> bool {
    buf.len() >= HEADER_LEN && (buf[1] & !FLAG_ZSTD) == 0 && buf[2] == 0 && buf[3] == 0
}

match decode_value(&raw) {
    Err(e) if e.to_string().contains("unknown flags") => {
        bail!("value written by an incompatible/newer encoder; upgrade this node");
    }
    other => other,
}

When it happens

Trigger: decode_value reading a buffer whose MAGIC matches but whose header flags contain bits other than FLAG_ZSTD in flag1, or any bits set in flag2/flag3 — e.g. values written by a newer encoder with extra flags.

Common situations: Reading meta data written by a newer Databend version that introduced a new header flag; bit-flip corruption in the header bytes; tests hand-assembling headers with wrong flag values.

Understand the failure class

Background: "cannot parse invalid wire-format data", "cannot unmarshal", "failed unmarshalling": protobuf unmarshal errors explained — this error's family across 10 libraries.

Related errors

• custom-encoded value too short to contain 4-byte header
• Invalid version
• Access denied: is outside allowed directories
• {e}
• e

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/3ea6a5a6485ecde8. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/api/src/kv/pb_api/compress.rs:134

pub fn decode_value(buf: &[u8]) -> Result<Cow<'_, [u8]>, io::Error> {
    if buf.is_empty() || buf[0] != MAGIC {
        return Ok(Cow::Borrowed(buf));
    }

    if buf.len() < HEADER_LEN {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            "custom-encoded value too short to contain 4-byte header",
        ));
    }

    let flag1 = buf[1];
    let flag2 = buf[2];
    let flag3 = buf[3];

    let unknown = (flag1 & !FLAG_ZSTD) | flag2 | flag3;
    if unknown != 0 {
        return Err(io::Error::new(
            io::ErrorKind::InvalidData,
            format!(
                "unknown flags in custom-encoded value header: [{:#04x}, {:#04x}, {:#04x}]",
                flag1, flag2, flag3
            ),
        ));
    }

    let payload = &buf[HEADER_LEN..];

    if flag1 & FLAG_ZSTD != 0 {
        let decompressed = zstd::decode_all(payload)?;
        return Ok(Cow::Owned(decompressed));
    }

    // No flags set: uncompressed custom format (reserved, not emitted by encode_value).
    Ok(Cow::Borrowed(payload))
}

View on GitHub (pinned to 288d84d76e)