{"record":{"id":"47695a5bc8e670bb","repo":"databendlabs/databend","slug":"custom-encoded-value-too-short-to-contain-4-byte-h","errorCode":null,"errorMessage":"custom-encoded value too short to contain 4-byte header","messagePattern":"custom-encoded value too short to contain 4-byte header","errorType":"validation","errorClass":"InvalidData","httpStatus":null,"severity":"error","filePath":"src/meta/api/src/kv/pb_api/compress.rs","lineNumber":122,"sourceCode":"\n/// Global encoder instance, used by [`codec::encode_pb`].\n///\n/// Compression is disabled by default. Call [`GLOBAL_ENCODER.set_compress(true)`]\n/// at startup to enable it.\npub static GLOBAL_ENCODER: Encoder = Encoder::new(false);\n\n/// Decode a value produced by [`Encoder::encode_value`] or a legacy raw protobuf value.\n///\n/// Returns `Cow::Borrowed` for legacy/uncompressed data (zero copy),\n/// and `Cow::Owned` for decompressed data.\n/// Returns an error if the header has unknown flag bits set (forward-compat guard).\npub fn decode_value(buf: &[u8]) -> Result<Cow<'_, [u8]>, io::Error> {\n    if buf.is_empty() || buf[0] != MAGIC {\n        return Ok(Cow::Borrowed(buf));\n    }\n\n    if buf.len() < HEADER_LEN {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            \"custom-encoded value too short to contain 4-byte header\",\n        ));\n    }\n\n    let flag1 = buf[1];\n    let flag2 = buf[2];\n    let flag3 = buf[3];\n\n    let unknown = (flag1 & !FLAG_ZSTD) | flag2 | flag3;\n    if unknown != 0 {\n        return Err(io::Error::new(\n            io::ErrorKind::InvalidData,\n            format!(\n                \"unknown flags in custom-encoded value header: [{:#04x}, {:#04x}, {:#04x}]\",\n                flag1, flag2, flag3\n            ),\n        ));","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/api/src/kv/pb_api/compress.rs#L104-L140","documentation":"decode_value in the meta kv pb_api implements a custom value encoding: a 1-byte MAGIC followed by a 4-byte header of flags. If a buffer starts with the MAGIC byte (so it claims to be custom-encoded) but is shorter than the 4-byte header, it is corrupt and decode_value returns io::ErrorKind::InvalidData with this message.","triggerScenarios":"Calling decode_value on a buffer whose first byte equals MAGIC but whose length is 1-3 bytes — i.e. a truncated or corrupted meta storage value, or a value whose payload coincidentally begins with the MAGIC byte and got truncated.","commonSituations":"Corrupted or truncated underlying KV storage records (e.g. partial write, disk issue); hand-crafted test bytes starting with the magic byte; restore/copy of meta data that was cut short.","solutions":["Treat the value as corrupt: re-write the key with a fresh encode_value and re-run the operation.","Verify the meta store's backing storage for truncation/corruption; restore from a snapshot/backup if records are damaged.","If the value legitimately starts with the magic byte, ensure it was produced by encode_value so the full header is present."],"exampleFix":null,"handlingStrategy":"type-guard","validationCode":"fn looks_like_custom_value(buf: &[u8]) -> bool {\n    buf.first() == Some(&MAGIC) && buf.len() >= HEADER_LEN\n}","typeGuard":null,"tryCatchPattern":"let value = match decode_value(&raw) {\n    Ok(v) => v,\n    Err(e) if e.kind() == io::ErrorKind::InvalidData => {\n        // treat as corrupt; rebuild the record from source of truth\n        reencode_and_store(key)?;\n        Cow::Owned(default_value())\n    }\n    Err(e) => return Err(e.into()),\n};","preventionTips":["Only feed decode_value bytes produced by encode_value.","Protect meta storage from partial writes; use fsync'd, atomic record writes.","Restore meta data only from complete, verified snapshots."],"tags":["meta-service","serialization","corruption","kv-storage"],"backgroundTag":"protobuf-unmarshal-failed","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}