{"record":{"id":"7971256f01213d7f","repo":"headroomlabs-ai/headroom","slug":"encoding-not-available-for-self-model-tokenize","errorCode":null,"errorMessage":"Encoding not available for {self.model} - tokenizer {self.tokenizer_name} could not be loaded","messagePattern":"Encoding not available for (.+?) - tokenizer (.+?) could not be loaded","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"warning","filePath":"headroom/tokenizers/huggingface.py","lineNumber":365,"sourceCode":"                # Fall back to base implementation\n                pass\n\n        return super().count_messages(messages)\n\n    def encode(self, text: str) -> list[int]:\n        \"\"\"Encode text to token IDs.\n\n        Args:\n            text: Text to encode.\n\n        Returns:\n            List of token IDs.\n\n        Raises:\n            NotImplementedError: If tokenizer not available.\n        \"\"\"\n        if self._use_fallback():\n            raise NotImplementedError(\n                f\"Encoding not available for {self.model} - \"\n                f\"tokenizer {self.tokenizer_name} could not be loaded\"\n            )\n        return self.tokenizer.encode(text, add_special_tokens=False)\n\n    def decode(self, tokens: list[int]) -> str:\n        \"\"\"Decode token IDs to text.\n\n        Args:\n            tokens: List of token IDs.\n\n        Returns:\n            Decoded text.\n\n        Raises:\n            NotImplementedError: If tokenizer not available.\n        \"\"\"\n        if self._use_fallback():","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/tokenizers/huggingface.py#L347-L383","documentation":"HuggingFaceTokenizer.encode() raises NotImplementedError when the underlying transformers tokenizer could not be loaded and the instance is running in fallback (count-only estimation) mode. The message names the model and the tokenizer that failed to load, distinguishing 'temporarily unavailable' from 'never supported'.","triggerScenarios":"Constructing HuggingFaceTokenizer for a model whose tokenizer files failed to download (offline env, HF hub blocked, bad model id) — _use_fallback() becomes true — then calling encode(); count() still works via estimation.","commonSituations":"CI without HF_TOKEN or with restricted egress to huggingface.co; typo'd or retired HF model ids; corporate proxies breaking huggingface_hub downloads; transient hub outages leaving no local cache.","solutions":["Fix the load: correct the model id, ensure network access / HF_TOKEN, or pre-download: `huggingface-cli download <model>` with HF_HOME set to a shared cache.","Run with HF_HUB_OFFLINE=1 only after warming the cache so is_available()/lazy load succeeds.","If encode is optional, catch NotImplementedError and degrade to count()-only behavior."],"exampleFix":"# before\ntok = HuggingFaceTokenizer(\"mistralai/Mistral-7B-v0.1\")  # load failed -> fallback\nids = tok.encode(text)  # NotImplementedError\n\n# after\n# pre-warm in a networked step: huggingface-cli download mistralai/Mistral-7B-v0.1\ntok = HuggingFaceTokenizer(\"mistralai/Mistral-7B-v0.1\")\nids = tok.encode(text)","handlingStrategy":"fallback","validationCode":"from headroom.tokenizers.huggingface import HuggingFaceTokenizer\nassert HuggingFaceTokenizer.is_available(), \"install transformers\"\n# probe load success without raising:\ntok = HuggingFaceTokenizer(model)\nassert not tok._use_fallback(), f\"tokenizer {tok.tokenizer_name} failed to load\"","typeGuard":"def hf_ready(tok) -> bool:\n    return not tok._use_fallback()","tryCatchPattern":"try:\n    ids = tok.encode(text)\nexcept NotImplementedError as e:\n    logger.warning(\"HF tokenizer in fallback mode: %s\", e)\n    ids = None","preventionTips":["Pre-download HF tokenizer files into HF_HOME during image build.","Set HF_TOKEN for gated models and verify egress to huggingface.co.","Probe _use_fallback() after construction and fail deployment checks early."],"tags":["tokenizer","huggingface","fallback","network","not-implemented"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}