can1357/oh-my-pi · error

utoken: zstd decode failed

Error message

utoken: zstd decode failed

What it means

The utoken tokenizer's RankTable::parse expects a zstd-compressed UTOK1 blob embedded at compile time. zstd::decode_all(...).expect() panics with this message when decompression fails — the blob is truncated, corrupted, or not zstd data. Like other embedded-blob asserts, this is treated as a build error, not a runtime input problem.

Solutions

1. Rebuild the native module cleanly so the embedded blob is regenerated
2. Verify the binary checksum and reinstall if corrupted
3. If you supply a custom UTOK blob, confirm it is valid zstd that decompresses to UTOK1-formatted payload
4. Report to maintainers if a stock binary reproduces this

try {
  tokens = native.tokenize(text);
} catch (err) {
  if (String(err?.message).includes('utoken: zstd decode failed')) {
    tokens = jsTokenizer.tokenize(text); // JS fallback
  } else { throw err; }
}

When it happens

Trigger: First use of a utoken tokenizer (triggering RankTable parsing) when the embedded vocabulary blob is corrupt — broken build/embedding, binary corruption, or a hand-patched vocab asset.

Common situations: Damaged binaries (bad download, aggressive post-processing); custom builds with a replaced or mis-compressed utok blob; toolchain changes that broke embedding of the asset.

Related errors

• bundled Silver.ttf must parse
• native task ` ` panicked
• Binary build did not produce
• bun pm pack failed
• Bundled agent " " for vibe cli " " is unavailable.

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/de78c8749870b4bd. Report an issue: GitHub.

Appendix: source

Thrown at crates/pi-natives/src/utok/bpe.rs:126

	/// absent (ranks are vocab indices, far below the sentinel).
	pairs:             Box<[u32; 65536]>,
	/// Tokens of 1 or 3..=15 bytes, keyed by [`pack`].
	short:             HashMap<u128, u32, Fx>,
	/// Tokens longer than 15 bytes.
	long:              FxMap,
	/// Longest token in bytes; callers may use it to bound scans.
	pub max_token_len: usize,
}

impl RankTable {
	/// Parse a zstd-compressed UTOK1 blob. Panics on malformed data — the
	/// blobs are compile-time embedded, so corruption is a build error.
	///
	/// Zero-length entries are *skipped*: packers emit merge-unreachable
	/// ("dead") vocab slots as empty strings to keep rank contiguity, and
	/// those ranks must never be produced.
	pub fn parse(zst: &[u8]) -> Self {
		let raw = zstd::decode_all(zst).expect("utoken: zstd decode failed");
		let mut p = &raw[..];
		assert_eq!(&p[..6], b"UTOK1\n", "utoken: bad magic");
		p = &p[6..];
		let n = u32::from_le_bytes(p[..4].try_into().unwrap()) as usize;
		p = &p[4..];
		let mut pairs: Box<[u32; 65536]> =
			vec![u32::MAX; 65536].into_boxed_slice().try_into().unwrap();
		let mut short = HashMap::with_capacity_and_hasher(n, Fx::default());
		let mut long = FxMap::default();
		let mut max_token_len = 0usize;
		for rank in 0..n as u32 {
			let mut len = 0usize;
			let mut shift = 0;
			loop {
				let b = p[0];
				p = &p[1..];
				len |= ((b & 0x7f) as usize) << shift;
				if b < 0x80 {

View on GitHub (pinned to 9690622007)