can1357/oh-my-pi · error

utoken: zstd decode failed

Error message

utoken: zstd decode failed

What it means

The utoken tokenizer's RankTable::parse expects a zstd-compressed UTOK1 blob embedded at compile time. zstd::decode_all(...).expect() panics with this message when decompression fails — the blob is truncated, corrupted, or not zstd data. Like other embedded-blob asserts, this is treated as a build error, not a runtime input problem.

Source

Thrown at crates/pi-natives/src/utok/bpe.rs:126

	/// absent (ranks are vocab indices, far below the sentinel).
	pairs:             Box<[u32; 65536]>,
	/// Tokens of 1 or 3..=15 bytes, keyed by [`pack`].
	short:             HashMap<u128, u32, Fx>,
	/// Tokens longer than 15 bytes.
	long:              FxMap,
	/// Longest token in bytes; callers may use it to bound scans.
	pub max_token_len: usize,
}

impl RankTable {
	/// Parse a zstd-compressed UTOK1 blob. Panics on malformed data — the
	/// blobs are compile-time embedded, so corruption is a build error.
	///
	/// Zero-length entries are *skipped*: packers emit merge-unreachable
	/// ("dead") vocab slots as empty strings to keep rank contiguity, and
	/// those ranks must never be produced.
	pub fn parse(zst: &[u8]) -> Self {
		let raw = zstd::decode_all(zst).expect("utoken: zstd decode failed");
		let mut p = &raw[..];
		assert_eq!(&p[..6], b"UTOK1\n", "utoken: bad magic");
		p = &p[6..];
		let n = u32::from_le_bytes(p[..4].try_into().unwrap()) as usize;
		p = &p[4..];
		let mut pairs: Box<[u32; 65536]> =
			vec![u32::MAX; 65536].into_boxed_slice().try_into().unwrap();
		let mut short = HashMap::with_capacity_and_hasher(n, Fx::default());
		let mut long = FxMap::default();
		let mut max_token_len = 0usize;
		for rank in 0..n as u32 {
			let mut len = 0usize;
			let mut shift = 0;
			loop {
				let b = p[0];
				p = &p[1..];
				len |= ((b & 0x7f) as usize) << shift;
				if b < 0x80 {

View on GitHub (pinned to 9690622007)

Solutions

1. Rebuild the native module cleanly so the embedded blob is regenerated
2. Verify the binary checksum and reinstall if corrupted
3. If you supply a custom UTOK blob, confirm it is valid zstd that decompresses to UTOK1-formatted payload
4. Report to maintainers if a stock binary reproduces this

try {
  tokens = native.tokenize(text);
} catch (err) {
  if (String(err?.message).includes('utoken: zstd decode failed')) {
    tokens = jsTokenizer.tokenize(text); // JS fallback
  } else { throw err; }
}

When it happens

Trigger: First use of a utoken tokenizer (triggering RankTable parsing) when the embedded vocabulary blob is corrupt — broken build/embedding, binary corruption, or a hand-patched vocab asset.

Common situations: Damaged binaries (bad download, aggressive post-processing); custom builds with a replaced or mis-compressed utok blob; toolchain changes that broke embedding of the asset.

Related errors

• bundled Silver.ttf must parse
• native task `{tag}` panicked: {message}
• modified change requires peer root
• native spelling thread stopped
• spelling range start is too large

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/de78c8749870b4bd. Report an issue: GitHub.