{"record":{"id":"6cf945ed61b4247c","repo":"Comfy-Org/ComfyUI","slug":"lens-tokenizer-requires-the-tokenizer-json-byt","errorCode":null,"errorMessage":"Lens tokenizer requires the ``tokenizer_json`` byte tensor in the encoder state dict. Re-bundle the encoder via bundle_te.py so it embeds the tokenizer.","messagePattern":"Lens tokenizer requires the ``tokenizer_json`` byte tensor in the encoder state dict\\. Re-bundle the encoder via bundle_te\\.py so it embeds the tokenizer\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"comfy/text_encoders/gpt_oss.py","lineNumber":432,"sourceCode":"\n# GPT-OSS-20B fixed token IDs (from the tokenizer's added-tokens table).\n_LENS_PAD_TOKEN_ID = 199999  # <|endoftext|>\n\n\nclass _GptOssRawTokenizer:\n    \"\"\"Raw ``tokenizers.Tokenizer`` wrapper.\n\n    The tokenizer JSON ships as a byte tensor inside the encoder checkpoint\n    (``tokenizer_json`` key) rather than as a committed file. Extracted\n    it in ``sd.py`` and passes it here via ``tokenizer_data``.\n    \"\"\"\n\n    def __init__(self, tokenizer_json_bytes=None, **kwargs):\n        from tokenizers import Tokenizer\n        if isinstance(tokenizer_json_bytes, torch.Tensor):\n            tokenizer_json_bytes = bytes(tokenizer_json_bytes.tolist())\n        if tokenizer_json_bytes is None:\n            raise ValueError(\n                \"Lens tokenizer requires the ``tokenizer_json`` byte tensor in the \"\n                \"encoder state dict. Re-bundle the encoder via bundle_te.py so it \"\n                \"embeds the tokenizer.\"\n            )\n        self.tokenizer = Tokenizer.from_str(tokenizer_json_bytes.decode(\"utf-8\"))\n\n    @classmethod\n    def from_pretrained(cls, tokenizer_data, **kwargs):\n        return cls(tokenizer_json_bytes=tokenizer_data, **kwargs)\n\n    def __call__(self, text):\n        return {\"input_ids\": self.tokenizer.encode(text, add_special_tokens=False).ids}\n\n    def get_vocab(self):\n        return self.tokenizer.get_vocab()\n\n    def convert_tokens_to_ids(self, tokens):\n        return [self.tokenizer.token_to_id(t) for t in tokens]","sourceCodeStart":414,"sourceCodeEnd":450,"githubUrl":"https://github.com/Comfy-Org/ComfyUI/blob/1c6d8d45b3693bfbb32385b410d813a7fd6be216/comfy/text_encoders/gpt_oss.py#L414-L450","documentation":"The gpt-oss 'Lens' text encoder ships its tokenizer as a byte tensor under the 'tokenizer_json' key inside the encoder safetensors checkpoint, not as a separate file. The wrapper class raises when tokenizer_data passed via from_pretrained is None, meaning the loaded state dict had no tokenizer_json entry. This happens when the encoder was bundled with an older/foreign script that omitted the tokenizer.","triggerScenarios":"Loading a gpt-oss encoder checkpoint that lacks the 'tokenizer_json' tensor in its state dict (e.g. re-saved weights from the original HF repo without running bundle_te.py), then instantiating the tokenizer via LensTokenizer.from_pretrained(tokenizer_data=None).","commonSituations":"User converted gpt-oss weights with an outdated conversion/bundling script; user downloaded a third-party re-pack of the encoder; the sd.py extraction step dropped the tokenizer_json key during state-dict filtering.","solutions":["Re-bundle the encoder with the repo's bundle_te.py so the tokenizer JSON is embedded as the 'tokenizer_json' byte tensor","Verify the checkpoint contains the key: load the safetensors and confirm 'tokenizer_json' exists before passing it to the TE","If bundling manually, serialize the tokenizer.json with tokenizers.Tokenizer.save_str/reads and store it as a uint8 tensor under 'tokenizer_json'"],"exampleFix":"// before\ncheckpoint = \"gpt-oss-te.safetensors\"  # bundled without tokenizer\n\n# after\n# re-run: python bundle_te.py --model gpt-oss-text-encoder --out gpt-oss-te.safetensors\nsd = safetensors.torch.load_file(\"gpt-oss-te.safetensors\")\nassert \"tokenizer_json\" in sd, \"re-bundle encoder via bundle_te.py\"","handlingStrategy":"validation","validationCode":"import safetensors.torch\nsd = safetensors.torch.load_file(encoder_path)\nif 'tokenizer_json' not in sd:\n    raise ValueError(f'{encoder_path} lacks embedded tokenizer; re-bundle with bundle_te.py')","typeGuard":"def has_embedded_tokenizer(sd: dict) -> bool:\n    v = sd.get('tokenizer_json')\n    return v is not None and (isinstance(v, (bytes, bytearray)) or getattr(v, 'dtype', None) is not None)","tryCatchPattern":null,"preventionTips":["Only load gpt-oss encoders bundled by the repo's bundle_te.py","Check for the tokenizer_json key immediately after loading, before constructing the TE"],"tags":["text-encoder","tokenizer","checkpoint","gpt-oss"],"backgroundTag":null,"analyzedSha":"1c6d8d45b3693bfbb32385b410d813a7fd6be216","analyzedAt":"2026-08-14T19:37:18.893Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}