{"record":{"id":"acfc7e20a6f69195","repo":"headroomlabs-ai/headroom","slug":"decoding-not-available-for-self-model-tokenize","errorCode":null,"errorMessage":"Decoding not available for {self.model} - tokenizer {self.tokenizer_name} could not be loaded","messagePattern":"Decoding not available for (.+?) - tokenizer (.+?) could not be loaded","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"warning","filePath":"headroom/tokenizers/huggingface.py","lineNumber":384,"sourceCode":"                f\"Encoding not available for {self.model} - \"\n                f\"tokenizer {self.tokenizer_name} could not be loaded\"\n            )\n        return self.tokenizer.encode(text, add_special_tokens=False)\n\n    def decode(self, tokens: list[int]) -> str:\n        \"\"\"Decode token IDs to text.\n\n        Args:\n            tokens: List of token IDs.\n\n        Returns:\n            Decoded text.\n\n        Raises:\n            NotImplementedError: If tokenizer not available.\n        \"\"\"\n        if self._use_fallback():\n            raise NotImplementedError(\n                f\"Decoding not available for {self.model} - \"\n                f\"tokenizer {self.tokenizer_name} could not be loaded\"\n            )\n        return self.tokenizer.decode(tokens)\n\n    @classmethod\n    def is_available(cls) -> bool:\n        \"\"\"Check if HuggingFace tokenizers are available.\n\n        Returns:\n            True if transformers is installed.\n        \"\"\"\n        try:\n            import transformers  # noqa: F401\n\n            return True\n        except ImportError:\n            return False","sourceCodeStart":366,"sourceCodeEnd":402,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/tokenizers/huggingface.py#L366-L402","documentation":"HuggingFaceTokenizer.decode() raises NotImplementedError when the transformers tokenizer failed to load and the instance is in estimation-fallback mode (checked via _use_fallback()). The message identifies the model and tokenizer name so you can tell which asset failed. Counting still works; token<->text round-trips do not.","triggerScenarios":"Same fallback condition as encode: tokenizer assets missing or unloadable (offline environment, bad model id, hub auth failure), followed by a call to decode(tokens).","commonSituations":"Air-gapped or proxied CI where huggingface_hub cannot fetch tokenizer.json; expired/gated-model access (HF_TOKEN missing for gated repos); cache corruption in HF_HOME.","solutions":["Restore the load: valid model id, credentials for gated models (HF_TOKEN), reachable hub or pre-populated HF_HOME cache.","Validate availability before decoding: HuggingFaceTokenizer.is_available() plus a check that _use_fallback() is False (or attempt a tiny encode as a probe).","Catch NotImplementedError and fall back to count-only logic if round-tripping is optional."],"exampleFix":"# before\ntext = hf_tok.decode(ids)  # NotImplementedError: tokenizer not loaded\n\n# after\nif not hf_tok._use_fallback():\n    text = hf_tok.decode(ids)\nelse:\n    raise RuntimeError(\"warm HF cache before running decode path\")","handlingStrategy":"fallback","validationCode":"tok = HuggingFaceTokenizer(model)\nif tok._use_fallback():\n    raise RuntimeError(\"HF tokenizer not loaded; warm cache or fix network before decode\")","typeGuard":"def hf_decode_ready(tok) -> bool:\n    return not tok._use_fallback()","tryCatchPattern":"try:\n    text = tok.decode(ids)\nexcept NotImplementedError as e:\n    logger.warning(\"decode unavailable (fallback mode): %s\", e)\n    text = \"\"","preventionTips":["Warm the HF cache in a networked setup step.","Check _use_fallback() before decode-dependent logic.","Monitor tokenizer load failures and alert rather than silently estimating."],"tags":["tokenizer","huggingface","fallback","network","not-implemented"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}