{"record":{"id":"de78c8749870b4bd","repo":"can1357/oh-my-pi","slug":"utoken-zstd-decode-failed","errorCode":null,"errorMessage":"utoken: zstd decode failed","messagePattern":"utoken: zstd decode failed","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/pi-natives/src/utok/bpe.rs","lineNumber":126,"sourceCode":"\t/// absent (ranks are vocab indices, far below the sentinel).\n\tpairs:             Box<[u32; 65536]>,\n\t/// Tokens of 1 or 3..=15 bytes, keyed by [`pack`].\n\tshort:             HashMap<u128, u32, Fx>,\n\t/// Tokens longer than 15 bytes.\n\tlong:              FxMap,\n\t/// Longest token in bytes; callers may use it to bound scans.\n\tpub max_token_len: usize,\n}\n\nimpl RankTable {\n\t/// Parse a zstd-compressed UTOK1 blob. Panics on malformed data — the\n\t/// blobs are compile-time embedded, so corruption is a build error.\n\t///\n\t/// Zero-length entries are *skipped*: packers emit merge-unreachable\n\t/// (\"dead\") vocab slots as empty strings to keep rank contiguity, and\n\t/// those ranks must never be produced.\n\tpub fn parse(zst: &[u8]) -> Self {\n\t\tlet raw = zstd::decode_all(zst).expect(\"utoken: zstd decode failed\");\n\t\tlet mut p = &raw[..];\n\t\tassert_eq!(&p[..6], b\"UTOK1\\n\", \"utoken: bad magic\");\n\t\tp = &p[6..];\n\t\tlet n = u32::from_le_bytes(p[..4].try_into().unwrap()) as usize;\n\t\tp = &p[4..];\n\t\tlet mut pairs: Box<[u32; 65536]> =\n\t\t\tvec![u32::MAX; 65536].into_boxed_slice().try_into().unwrap();\n\t\tlet mut short = HashMap::with_capacity_and_hasher(n, Fx::default());\n\t\tlet mut long = FxMap::default();\n\t\tlet mut max_token_len = 0usize;\n\t\tfor rank in 0..n as u32 {\n\t\t\tlet mut len = 0usize;\n\t\t\tlet mut shift = 0;\n\t\t\tloop {\n\t\t\t\tlet b = p[0];\n\t\t\t\tp = &p[1..];\n\t\t\t\tlen |= ((b & 0x7f) as usize) << shift;\n\t\t\t\tif b < 0x80 {","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/crates/pi-natives/src/utok/bpe.rs#L108-L144","documentation":"The utoken tokenizer's RankTable::parse expects a zstd-compressed UTOK1 blob embedded at compile time. zstd::decode_all(...).expect() panics with this message when decompression fails — the blob is truncated, corrupted, or not zstd data. Like other embedded-blob asserts, this is treated as a build error, not a runtime input problem.","triggerScenarios":"First use of a utoken tokenizer (triggering RankTable parsing) when the embedded vocabulary blob is corrupt — broken build/embedding, binary corruption, or a hand-patched vocab asset.","commonSituations":"Damaged binaries (bad download, aggressive post-processing); custom builds with a replaced or mis-compressed utok blob; toolchain changes that broke embedding of the asset.","solutions":["Rebuild the native module cleanly so the embedded blob is regenerated","Verify the binary checksum and reinstall if corrupted","If you supply a custom UTOK blob, confirm it is valid zstd that decompresses to UTOK1-formatted payload","Report to maintainers if a stock binary reproduces this"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  tokens = native.tokenize(text);\n} catch (err) {\n  if (String(err?.message).includes('utoken: zstd decode failed')) {\n    tokens = jsTokenizer.tokenize(text); // JS fallback\n  } else { throw err; }\n}","preventionTips":["Verify binary integrity after download/install","Do not hand-edit embedded vocab blobs; use the packer tooling","Rebuild from source if embedding was customized"],"tags":["native","tokenizer","zstd","panic","build"],"backgroundTag":"embedded-asset-corrupt","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}