huggingface/tokenizers · error

Helper

Error message

Helper

What it means

During serde deserialization of a Decoder enum, an internal DecoderHelper is used to capture the tagged variant plus remaining fields; .expect("Helper") panics if that helper deserialization fails. Because the helper mirrors the enum's variants, this effectively means the decoder definition in the serialized tokenizer (e.g. tokenizer.json) does not match any known decoder variant. The library throws it as an invariant guard while converting the tagged representation back into a typed decoder.

Solutions

1. Upgrade the tokenizers library to a version that recognizes the decoder "type" in your tokenizer.json (align versions with whatever tool produced the file).
2. Validate tokenizer.json locally: check decoders.type is one of the supported values (BPE, WordPiece, WordLevel, Metaspace, CTC, Sequence, Replace, Fuse, Strip, ByteFallback, ...) and fix typos.
3. Regenerate the tokenizer file with a matching tokenizers version rather than hand-editing it.

Example fix

// before (tokenizer.json)
"decoders": { "type": "FancyNewDecoder", ... }

// after — either upgrade the library, or use a supported type
"decoders": { "type": "Metaspace", "replacement": "▁", "prepend_scheme": "always" }

// validate decoder type before deserializing the tokenizer
const SUPPORTED = ['BPE','WordPiece','WordLevel','Metaspace','CTC','Sequence','Replace','Fuse','Strip','ByteFallback','BPEDecoder'];
if (cfg.decoders && !SUPPORTED.includes(cfg.decoders.type) && cfg.decoders.type !== 'Sequence') {
  throw new Error(`Unsupported decoder type: ${cfg.decoders.type}; upgrade tokenizers`);
}

function hasKnownDecoder(cfg) { return !cfg.decoders || SUPPORTED_DECODER_TYPES.has(cfg.decoders.type); }

try { tok = Tokenizer.fromJSON(json); } catch (e) { /* inspect decoders.type, upgrade tokenizers or strip decoder */ }

When it happens

Trigger: Deserializing a tokenizer JSON whose decoders.type value is unknown/unrecognized (or whose tagged payload fails to deserialize into any DecoderHelper variant), e.g. loading a tokenizer file produced by a newer tokenizers version into an older one.

Common situations: Version mismatch: tokenizer.json written by a newer tokenizers release with a decoder type the running library doesn't support; hand-edited tokenizer.json with a typo'd "type" field; corrupted or truncated JSON payload.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• NormalizedString bad split
• AddedVocabulary bad split
• Native binding package version mismatch, expected…
• sep_token not found in the vocabulary
• cls_token not found in the vocabulary

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/545f60071d1dcb47. Report an issue: GitHub.

Appendix: source

Thrown at tokenizers/src/decoders/mod.rs:90

            Legacy(serde_json::Value),
        }

        #[derive(Deserialize)]
        #[serde(untagged)]
        pub enum DecoderUntagged {
            BPE(BPEDecoder),
            ByteLevel(ByteLevel),
            WordPiece(WordPiece),
            Metaspace(Metaspace),
            CTC(CTC),
            Sequence(Sequence),
            Replace(Replace),
            Fuse(Fuse),
            Strip(Strip),
            ByteFallback(ByteFallback),
        }

        let helper = DecoderHelper::deserialize(deserializer).expect("Helper");
        Ok(match helper {
            DecoderHelper::Tagged(model) => {
                let mut values: serde_json::Map<String, serde_json::Value> =
                    serde_json::from_value(model.rest).map_err(serde::de::Error::custom)?;
                values.insert(
                    "type".to_string(),
                    serde_json::to_value(&model.variant).map_err(serde::de::Error::custom)?,
                );
                let values = serde_json::Value::Object(values);
                match model.variant {
                    EnumType::BPEDecoder => DecoderWrapper::BPE(
                        serde_json::from_value(values).map_err(serde::de::Error::custom)?,
                    ),
                    EnumType::ByteLevel => DecoderWrapper::ByteLevel(
                        serde_json::from_value(values).map_err(serde::de::Error::custom)?,
                    ),
                    EnumType::WordPiece => DecoderWrapper::WordPiece(
                        serde_json::from_value(values).map_err(serde::de::Error::custom)?,

View on GitHub (pinned to 6cfd9d385c)