Comfy-Org/ComfyUI · error · ValueError

Lens tokenizer requires the ``tokenizer_json`` byte tensor i

Error message

Lens tokenizer requires the ``tokenizer_json`` byte tensor in the encoder state dict. Re-bundle the encoder via bundle_te.py so it embeds the tokenizer.

What it means

The gpt-oss 'Lens' text encoder ships its tokenizer as a byte tensor under the 'tokenizer_json' key inside the encoder safetensors checkpoint, not as a separate file. The wrapper class raises when tokenizer_data passed via from_pretrained is None, meaning the loaded state dict had no tokenizer_json entry. This happens when the encoder was bundled with an older/foreign script that omitted the tokenizer.

Source

Thrown at comfy/text_encoders/gpt_oss.py:432

# GPT-OSS-20B fixed token IDs (from the tokenizer's added-tokens table).
_LENS_PAD_TOKEN_ID = 199999  # <|endoftext|>


class _GptOssRawTokenizer:
    """Raw ``tokenizers.Tokenizer`` wrapper.

    The tokenizer JSON ships as a byte tensor inside the encoder checkpoint
    (``tokenizer_json`` key) rather than as a committed file. Extracted
    it in ``sd.py`` and passes it here via ``tokenizer_data``.
    """

    def __init__(self, tokenizer_json_bytes=None, **kwargs):
        from tokenizers import Tokenizer
        if isinstance(tokenizer_json_bytes, torch.Tensor):
            tokenizer_json_bytes = bytes(tokenizer_json_bytes.tolist())
        if tokenizer_json_bytes is None:
            raise ValueError(
                "Lens tokenizer requires the ``tokenizer_json`` byte tensor in the "
                "encoder state dict. Re-bundle the encoder via bundle_te.py so it "
                "embeds the tokenizer."
            )
        self.tokenizer = Tokenizer.from_str(tokenizer_json_bytes.decode("utf-8"))

    @classmethod
    def from_pretrained(cls, tokenizer_data, **kwargs):
        return cls(tokenizer_json_bytes=tokenizer_data, **kwargs)

    def __call__(self, text):
        return {"input_ids": self.tokenizer.encode(text, add_special_tokens=False).ids}

    def get_vocab(self):
        return self.tokenizer.get_vocab()

    def convert_tokens_to_ids(self, tokens):
        return [self.tokenizer.token_to_id(t) for t in tokens]

View on GitHub (pinned to 1c6d8d45b3)

Solutions

1. Re-bundle the encoder with the repo's bundle_te.py so the tokenizer JSON is embedded as the 'tokenizer_json' byte tensor
2. Verify the checkpoint contains the key: load the safetensors and confirm 'tokenizer_json' exists before passing it to the TE
3. If bundling manually, serialize the tokenizer.json with tokenizers.Tokenizer.save_str/reads and store it as a uint8 tensor under 'tokenizer_json'

Example fix

// before
checkpoint = "gpt-oss-te.safetensors"  # bundled without tokenizer

# after
# re-run: python bundle_te.py --model gpt-oss-text-encoder --out gpt-oss-te.safetensors
sd = safetensors.torch.load_file("gpt-oss-te.safetensors")
assert "tokenizer_json" in sd, "re-bundle encoder via bundle_te.py"

import safetensors.torch
sd = safetensors.torch.load_file(encoder_path)
if 'tokenizer_json' not in sd:
    raise ValueError(f'{encoder_path} lacks embedded tokenizer; re-bundle with bundle_te.py')

def has_embedded_tokenizer(sd: dict) -> bool:
    v = sd.get('tokenizer_json')
    return v is not None and (isinstance(v, (bytes, bytearray)) or getattr(v, 'dtype', None) is not None)

When it happens

Trigger: Loading a gpt-oss encoder checkpoint that lacks the 'tokenizer_json' tensor in its state dict (e.g. re-saved weights from the original HF repo without running bundle_te.py), then instantiating the tokenizer via LensTokenizer.from_pretrained(tokenizer_data=None).

Common situations: User converted gpt-oss weights with an outdated conversion/bundling script; user downloaded a third-party re-pack of the encoder; the sd.py extraction step dropped the tokenizer_json key during state-dict filtering.

Related errors

• MiniMax Music3 text encoder checkpoint is missing tokenizer_
• MiniMax Music3 tokenizer mismatch for {token}
• invalid tokenizer
• ERROR: clip input is invalid: None\n\nIf the clip is from a
• Unexpected token width: {out_x.shape[-1]}

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/6cf945ed61b4247c. Report an issue: GitHub.