zylon-ai/private-gpt · error · ValueError

Failed to load tokenizer: {e}

Error message

Failed to load tokenizer: {e}

What it means

Any non-OSError exception while loading the tokenizer/processor (bad tokenizer.json, incompatible transformers version, corrupted cache, permission errors) is wrapped as ValueError('Failed to load tokenizer: {e}') with the original exception chained. It is the catch-all branch of HuggingFaceTokenizer.from_pretrained.

Source

Thrown at private_gpt/components/llm/tokenizers/huggingface.py:106

            tokenizer: PreTrainedTokenizerBase
            if hasattr(loaded, "tokenizer"):
                processor = cast(ProcessorMixin, loaded)
                tokenizer = cast(PreTrainedTokenizerBase, loaded.tokenizer)
                is_multimodal = True
            else:
                tokenizer = cast(PreTrainedTokenizerBase, loaded)

            return cls(tokenizer, is_multimodal=is_multimodal, processor=processor)

        except OSError as e:
            if local_files_only:
                raise FileNotFoundError(
                    f"Local model files not found at '{model_id}'. "
                    f"Ensure the model is downloaded locally."
                ) from e
            raise ValueError(f"Could not load tokenizer from '{model_id}': {e}") from e
        except Exception as e:
            raise ValueError(f"Failed to load tokenizer: {e}") from e

    @classmethod
    def is_available(cls, model_id: str | Path | None, **kwargs: Any) -> bool:
        return bool(model_id)

    @property
    def all_special_tokens(self) -> list[str]:
        tokens: list[str] = self._tokenizer.all_special_tokens
        return tokens

    @property
    def all_special_ids(self) -> list[int]:
        ids: list[int] = self._tokenizer.all_special_ids
        return ids

    @property
    def bos_token_id(self) -> int:
        return cast(int, self._tokenizer.bos_token_id)

View on GitHub (pinned to 4a030776a3)

Solutions

1. Read the chained cause (`raise ... from e` — inspect __cause__) to identify the real failure and address it (fix permissions, re-download, etc.).
2. Clear the corrupted cache entry: rm -rf ~/.cache/huggingface/hub/models--<org>--<model> and retry.
3. Upgrade/downgrade transformers to a version compatible with the model's tokenizer files.

Example fix

# before: corrupted cache causes generic 'Failed to load tokenizer'
# after
rm -rf ~/.cache/huggingface/hub/models--mistralai--Mistral-7B-Instruct-v0.3
tok = HuggingFaceTokenizer.from_pretrained('mistralai/Mistral-7B-Instruct-v0.3')

try:
    tok = HuggingFaceTokenizer.from_pretrained(model_id)
except ValueError as e:
    if 'Failed to load tokenizer' in str(e):
        logger.error('tokenizer load failed, cause: %r', e.__cause__)
        # clear cache and retry once, then give up with a clear error
        raise

When it happens

Trigger: Corrupted HF cache entries; tokenizer files requiring a newer/older transformers version than installed (e.g. a new chat-template construct raising in AutoTokenizer); JSONDecodeError from a truncated download; PermissionError on cache dirs.

Common situations: Version drift between transformers and recently published models; interrupted downloads leaving partial files; read-only volumes for the cache; pickled/tokenizer artifacts needing trust_remote_code that was left off.

Related errors

• Unknown message role {self.role}. Expected 'system', 'user',
• Image size {image_size} exceeds maximum allowed size of {set
• Audio size {audio_size} exceeds maximum allowed size of {set
• Invalid tool specification: {tool}
• Invalid system item: {item}

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/2d2bb1183f111911. Report an issue: GitHub.