{"record":{"id":"545f60071d1dcb47","repo":"huggingface/tokenizers","slug":"helper","errorCode":null,"errorMessage":"Helper","messagePattern":"Helper","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"tokenizers/src/decoders/mod.rs","lineNumber":90,"sourceCode":"            Legacy(serde_json::Value),\n        }\n\n        #[derive(Deserialize)]\n        #[serde(untagged)]\n        pub enum DecoderUntagged {\n            BPE(BPEDecoder),\n            ByteLevel(ByteLevel),\n            WordPiece(WordPiece),\n            Metaspace(Metaspace),\n            CTC(CTC),\n            Sequence(Sequence),\n            Replace(Replace),\n            Fuse(Fuse),\n            Strip(Strip),\n            ByteFallback(ByteFallback),\n        }\n\n        let helper = DecoderHelper::deserialize(deserializer).expect(\"Helper\");\n        Ok(match helper {\n            DecoderHelper::Tagged(model) => {\n                let mut values: serde_json::Map<String, serde_json::Value> =\n                    serde_json::from_value(model.rest).map_err(serde::de::Error::custom)?;\n                values.insert(\n                    \"type\".to_string(),\n                    serde_json::to_value(&model.variant).map_err(serde::de::Error::custom)?,\n                );\n                let values = serde_json::Value::Object(values);\n                match model.variant {\n                    EnumType::BPEDecoder => DecoderWrapper::BPE(\n                        serde_json::from_value(values).map_err(serde::de::Error::custom)?,\n                    ),\n                    EnumType::ByteLevel => DecoderWrapper::ByteLevel(\n                        serde_json::from_value(values).map_err(serde::de::Error::custom)?,\n                    ),\n                    EnumType::WordPiece => DecoderWrapper::WordPiece(\n                        serde_json::from_value(values).map_err(serde::de::Error::custom)?,","sourceCodeStart":72,"sourceCodeEnd":108,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/tokenizers/src/decoders/mod.rs#L72-L108","documentation":"During serde deserialization of a Decoder enum, an internal DecoderHelper is used to capture the tagged variant plus remaining fields; .expect(\"Helper\") panics if that helper deserialization fails. Because the helper mirrors the enum's variants, this effectively means the decoder definition in the serialized tokenizer (e.g. tokenizer.json) does not match any known decoder variant. The library throws it as an invariant guard while converting the tagged representation back into a typed decoder.","triggerScenarios":"Deserializing a tokenizer JSON whose decoders.type value is unknown/unrecognized (or whose tagged payload fails to deserialize into any DecoderHelper variant), e.g. loading a tokenizer file produced by a newer tokenizers version into an older one.","commonSituations":"Version mismatch: tokenizer.json written by a newer tokenizers release with a decoder type the running library doesn't support; hand-edited tokenizer.json with a typo'd \"type\" field; corrupted or truncated JSON payload.","solutions":["Upgrade the tokenizers library to a version that recognizes the decoder \"type\" in your tokenizer.json (align versions with whatever tool produced the file).","Validate tokenizer.json locally: check decoders.type is one of the supported values (BPE, WordPiece, WordLevel, Metaspace, CTC, Sequence, Replace, Fuse, Strip, ByteFallback, ...) and fix typos.","Regenerate the tokenizer file with a matching tokenizers version rather than hand-editing it."],"exampleFix":"// before (tokenizer.json)\n\"decoders\": { \"type\": \"FancyNewDecoder\", ... }\n\n// after — either upgrade the library, or use a supported type\n\"decoders\": { \"type\": \"Metaspace\", \"replacement\": \"▁\", \"prepend_scheme\": \"always\" }","handlingStrategy":"validation","validationCode":"// validate decoder type before deserializing the tokenizer\nconst SUPPORTED = ['BPE','WordPiece','WordLevel','Metaspace','CTC','Sequence','Replace','Fuse','Strip','ByteFallback','BPEDecoder'];\nif (cfg.decoders && !SUPPORTED.includes(cfg.decoders.type) && cfg.decoders.type !== 'Sequence') {\n  throw new Error(`Unsupported decoder type: ${cfg.decoders.type}; upgrade tokenizers`);\n}","typeGuard":"function hasKnownDecoder(cfg) { return !cfg.decoders || SUPPORTED_DECODER_TYPES.has(cfg.decoders.type); }","tryCatchPattern":"try { tok = Tokenizer.fromJSON(json); } catch (e) { /* inspect decoders.type, upgrade tokenizers or strip decoder */ }","preventionTips":["Pin tokenizers version to match whatever generated tokenizer.json","Never hand-edit tokenizer.json decoder sections","Validate the JSON schema of tokenizer files before loading"],"tags":["serde","deserialization","tokenizer","version-mismatch","rust"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}