{"record":{"id":"3ea6a5a6485ecde8","repo":"databendlabs/databend","slug":"unknown-flags-in-custom-encoded-value-header","errorCode":null,"errorMessage":"unknown flags in custom-encoded value header: [{:#04x}, {:#04x}, {:#04x}]","messagePattern":"unknown flags in custom-encoded value header: \\[(.+?), (.+?), (.+?)\\]","errorType":"validation","errorClass":"InvalidData","httpStatus":null,"severity":"error","filePath":"src/meta/api/src/kv/pb_api/compress.rs","lineNumber":134,"sourceCode":"pub fn decode_value(buf: &[u8]) -> Result<Cow<'_, [u8]>, io::Error> {\n    if buf.is_empty() || buf[0] != MAGIC {\n        return Ok(Cow::Borrowed(buf));\n    }\n\n    if buf.len() < HEADER_LEN {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            \"custom-encoded value too short to contain 4-byte header\",\n        ));\n    }\n\n    let flag1 = buf[1];\n    let flag2 = buf[2];\n    let flag3 = buf[3];\n\n    let unknown = (flag1 & !FLAG_ZSTD) | flag2 | flag3;\n    if unknown != 0 {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            format!(\n                \"unknown flags in custom-encoded value header: [{:#04x}, {:#04x}, {:#04x}]\",\n                flag1, flag2, flag3\n            ),\n        ));\n    }\n\n    let payload = &buf[HEADER_LEN..];\n\n    if flag1 & FLAG_ZSTD != 0 {\n        let decompressed = zstd::decode_all(payload)?;\n        return Ok(Cow::Owned(decompressed));\n    }\n\n    // No flags set: uncompressed custom format (reserved, not emitted by encode_value).\n    Ok(Cow::Borrowed(payload))\n}","sourceCodeStart":116,"sourceCodeEnd":152,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/api/src/kv/pb_api/compress.rs#L116-L152","documentation":"The custom value header reserves flag1 bit FLAG_ZSTD, while flag2 and flag3 are entirely unused. decode_value ORs together the unknown bits of all three flag bytes and rejects the value with InvalidData when any is set, so values encoded by a future/unknown encoder version are refused instead of being misinterpreted.","triggerScenarios":"decode_value reading a buffer whose MAGIC matches but whose header flags contain bits other than FLAG_ZSTD in flag1, or any bits set in flag2/flag3 — e.g. values written by a newer encoder with extra flags.","commonSituations":"Reading meta data written by a newer Databend version that introduced a new header flag; bit-flip corruption in the header bytes; tests hand-assembling headers with wrong flag values.","solutions":["Identify the writer that set the unknown flags (likely a newer version) and upgrade the reader's Databend version to match.","Re-encode the value with the current encode_value so only supported flags are set.","Check the stored bytes for corruption (compare against a backup/snapshot) if no version change occurred."],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"fn header_flags_known(buf: &[u8]) -> bool {\n    buf.len() >= HEADER_LEN && (buf[1] & !FLAG_ZSTD) == 0 && buf[2] == 0 && buf[3] == 0\n}","typeGuard":null,"tryCatchPattern":"match decode_value(&raw) {\n    Err(e) if e.to_string().contains(\"unknown flags\") => {\n        bail!(\"value written by an incompatible/newer encoder; upgrade this node\");\n    }\n    other => other,\n}","preventionTips":["Upgrade reader nodes before any writer node that may set new header flags.","Keep HEADER_LEN/flag semantics documented so tests don't invent flags.","Run round-trip decode tests against values from all supported encoder versions."],"tags":["meta-service","serialization","forward-compatibility","kv-storage"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}