{"record":{"id":"3f674801fbdc6d25","repo":"zylon-ai/private-gpt","slug":"could-not-load-tokenizer-from-model-id-e","errorCode":null,"errorMessage":"Could not load tokenizer from '{model_id}': {e}","messagePattern":"Could not load tokenizer from '(.+?)': (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/huggingface.py","lineNumber":104,"sourceCode":"\n            # Extract tokenizer from multimodal processor if needed\n            tokenizer: PreTrainedTokenizerBase\n            if hasattr(loaded, \"tokenizer\"):\n                processor = cast(ProcessorMixin, loaded)\n                tokenizer = cast(PreTrainedTokenizerBase, loaded.tokenizer)\n                is_multimodal = True\n            else:\n                tokenizer = cast(PreTrainedTokenizerBase, loaded)\n\n            return cls(tokenizer, is_multimodal=is_multimodal, processor=processor)\n\n        except OSError as e:\n            if local_files_only:\n                raise FileNotFoundError(\n                    f\"Local model files not found at '{model_id}'. \"\n                    f\"Ensure the model is downloaded locally.\"\n                ) from e\n            raise ValueError(f\"Could not load tokenizer from '{model_id}': {e}\") from e\n        except Exception as e:\n            raise ValueError(f\"Failed to load tokenizer: {e}\") from e\n\n    @classmethod\n    def is_available(cls, model_id: str | Path | None, **kwargs: Any) -> bool:\n        return bool(model_id)\n\n    @property\n    def all_special_tokens(self) -> list[str]:\n        tokens: list[str] = self._tokenizer.all_special_tokens\n        return tokens\n\n    @property\n    def all_special_ids(self) -> list[int]:\n        ids: list[int] = self._tokenizer.all_special_ids\n        return ids\n\n    @property","sourceCodeStart":86,"sourceCodeEnd":122,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/huggingface.py#L86-L122","documentation":"With downloads allowed, any OSError from AutoProcessor.from_pretrained is wrapped as ValueError(\"Could not load tokenizer from '{model_id}': {e}\"). OSError from the Hub typically means the repo id is invalid/not found, the connection failed, or the repo is gated.","triggerScenarios":"HuggingFaceTokenizer.from_pretrained('nonexistent/model') with local_files_only=False; network outage or blocked HF endpoint in sandboxes; gated repo without prior `hf auth login`; malformed repo id (more than one slash, spaces).","commonSituations":"Typo'd or retired model ids; egress-restricted corporate networks; missing HF token for Llama-style gated models; proxies/SSL interception breaking huggingface_hub requests.","solutions":["Check the underlying message chained in {e}: fix the repo id or network access accordingly.","For gated models run `hf auth login` and set HF_TOKEN so the download is authorized.","For offline-capable setups, pre-download the model and pass local_files_only=True (which then gives the more specific FileNotFoundError if still missing)."],"exampleFix":"# before\ntok = HuggingFaceTokenizer.from_pretrained('mistralai/mistral-7b-typo')\n\n# after\nhf auth login  # if gated\ntok = HuggingFaceTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.3')","handlingStrategy":"try-catch","validationCode":"from huggingface_hub import HfApi\n\ndef repo_exists(model_id: str) -> bool:\n    try:\n        HfApi().repo_info(model_id)\n        return True\n    except Exception:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    tok = HuggingFaceTokenizer.from_pretrained(model_id)\nexcept ValueError as e:\n    if 'Could not load tokenizer' in str(e):\n        cause = e.__cause__\n        logger.error('load failed for %s: %s', model_id, cause)\n        raise ModelLoadError(model_id, cause) from e\n    raise","preventionTips":["Validate repo ids (org/name, exactly one slash) before passing them in.","Authenticate with `hf auth login` for gated models.","Log the chained cause — the underlying OSError carries the real reason."],"tags":["huggingface","network","model-id","auth"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}