{"record":{"id":"2d2bb1183f111911","repo":"zylon-ai/private-gpt","slug":"failed-to-load-tokenizer-e","errorCode":null,"errorMessage":"Failed to load tokenizer: {e}","messagePattern":"Failed to load tokenizer: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/huggingface.py","lineNumber":106,"sourceCode":"            tokenizer: PreTrainedTokenizerBase\n            if hasattr(loaded, \"tokenizer\"):\n                processor = cast(ProcessorMixin, loaded)\n                tokenizer = cast(PreTrainedTokenizerBase, loaded.tokenizer)\n                is_multimodal = True\n            else:\n                tokenizer = cast(PreTrainedTokenizerBase, loaded)\n\n            return cls(tokenizer, is_multimodal=is_multimodal, processor=processor)\n\n        except OSError as e:\n            if local_files_only:\n                raise FileNotFoundError(\n                    f\"Local model files not found at '{model_id}'. \"\n                    f\"Ensure the model is downloaded locally.\"\n                ) from e\n            raise ValueError(f\"Could not load tokenizer from '{model_id}': {e}\") from e\n        except Exception as e:\n            raise ValueError(f\"Failed to load tokenizer: {e}\") from e\n\n    @classmethod\n    def is_available(cls, model_id: str | Path | None, **kwargs: Any) -> bool:\n        return bool(model_id)\n\n    @property\n    def all_special_tokens(self) -> list[str]:\n        tokens: list[str] = self._tokenizer.all_special_tokens\n        return tokens\n\n    @property\n    def all_special_ids(self) -> list[int]:\n        ids: list[int] = self._tokenizer.all_special_ids\n        return ids\n\n    @property\n    def bos_token_id(self) -> int:\n        return cast(int, self._tokenizer.bos_token_id)","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/huggingface.py#L88-L124","documentation":"Any non-OSError exception while loading the tokenizer/processor (bad tokenizer.json, incompatible transformers version, corrupted cache, permission errors) is wrapped as ValueError('Failed to load tokenizer: {e}') with the original exception chained. It is the catch-all branch of HuggingFaceTokenizer.from_pretrained.","triggerScenarios":"Corrupted HF cache entries; tokenizer files requiring a newer/older transformers version than installed (e.g. a new chat-template construct raising in AutoTokenizer); JSONDecodeError from a truncated download; PermissionError on cache dirs.","commonSituations":"Version drift between transformers and recently published models; interrupted downloads leaving partial files; read-only volumes for the cache; pickled/tokenizer artifacts needing trust_remote_code that was left off.","solutions":["Read the chained cause (`raise ... from e` — inspect __cause__) to identify the real failure and address it (fix permissions, re-download, etc.).","Clear the corrupted cache entry: rm -rf ~/.cache/huggingface/hub/models--<org>--<model> and retry.","Upgrade/downgrade transformers to a version compatible with the model's tokenizer files."],"exampleFix":"# before: corrupted cache causes generic 'Failed to load tokenizer'\n# after\nrm -rf ~/.cache/huggingface/hub/models--mistralai--Mistral-7B-Instruct-v0.3\ntok = HuggingFaceTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.3')","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try:\n    tok = HuggingFaceTokenizer.from_pretrained(model_id)\nexcept ValueError as e:\n    if 'Failed to load tokenizer' in str(e):\n        logger.error('tokenizer load failed, cause: %r', e.__cause__)\n        # clear cache and retry once, then give up with a clear error\n        raise","preventionTips":["Always inspect __cause__ before guessing at fixes.","Keep transformers versions in sync with the models you pull.","Treat partially downloaded caches as suspect after any interrupted run; clean and re-download."],"tags":["huggingface","cache-corruption","versioning","wrapping"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}