{"record":{"id":"7e2e0fff82db4579","repo":"zylon-ai/private-gpt","slug":"huggingfacetokenizer-is-not-available-with-the-giv","errorCode":null,"errorMessage":"HuggingFaceTokenizer is not available with the given configuration.","messagePattern":"HuggingFaceTokenizer is not available with the given configuration\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/registry.py","lineNumber":59,"sourceCode":"    return EstimatorTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_remote_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.remote import RemoteTokenizeTokenizer\n\n    if not RemoteTokenizeTokenizer.is_available(**kwargs):\n        raise ValueError(\n            \"RemoteTokenizeTokenizer is not available with the given configuration.\"\n        )\n\n    return RemoteTokenizeTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_huggingface_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.huggingface import HuggingFaceTokenizer\n\n    if not HuggingFaceTokenizer.is_available(**kwargs):\n        raise ImportError(\n            \"HuggingFaceTokenizer is not available with the given configuration.\"\n        )\n\n    return HuggingFaceTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_default_tokenizer(**kwargs: Any) -> TokenizerBase:\n    # 1. HF as initial tokenizer\n    try:\n        return _build_huggingface_tokenizer(**kwargs)\n    except (ImportError, Exception):\n        pass\n\n    # 2. Use the remote tokenizer\n    try:\n        return _build_remote_tokenizer(**kwargs)\n    except (ImportError, ValueError, Exception):\n        pass","sourceCodeStart":41,"sourceCodeEnd":77,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/registry.py#L41-L77","documentation":"Raised by _build_huggingface_tokenizer when HuggingFaceTokenizer.is_available(**kwargs) returns False. It is an ImportError, signalling that the local HuggingFace tokenizer path cannot be used — most often because the 'transformers' dependency (huggingface extra) is not installed, or the required model/config cannot be resolved. Note that the default-tokenizer chain in the same file catches ImportError (and broadly Exception) and falls through to the next builder, so users normally only see this when explicitly requesting the huggingface mode.","triggerScenarios":"Explicitly requesting tokenizer mode 'huggingface' via get_tokenizer('huggingface', ...) when the huggingface extra is absent, or when is_available fails for the given model_id/config (missing transformers/tokenizers packages, unreachable Hub, gated model without credentials).","commonSituations":"Slim installation without the huggingface extra (uv sync --inexact --extra huggingface missing); CI environments with no network access to huggingface.co; gated/private models without a valid HF token; dependency upgrades dropping transformers.","solutions":["Install the huggingface extra: uv sync --inexact --extra huggingface (or pip install transformers tokenizers).","If the model is gated/private, set a valid HF_TOKEN and confirm the model_id is accessible.","Check the fallback chain: if you do not specifically need HF, use 'estimator' or 'tiktoken' mode which need no Hub access.","If is_available still fails, run HuggingFaceTokenizer.is_available(**kwargs) directly to see which dependency/config check rejects it."],"exampleFix":"# before\ntokenizer = get_tokenizer('huggingface')  # ImportError\n\n# after (shell)\n# uv sync --inexact --extra huggingface\ntokenizer = get_tokenizer('huggingface')","handlingStrategy":"validation","validationCode":"from private_gpt.components.llm.tokenizers.huggingface import HuggingFaceTokenizer\n\nif not HuggingFaceTokenizer.is_available(**kwargs):\n    raise SystemExit('Install the huggingface extra or switch tokenizer_mode')","typeGuard":null,"tryCatchPattern":"try:\n    tok = get_tokenizer('huggingface', **kwargs)\nexcept ImportError:\n    tok = get_tokenizer('estimator', **kwargs)","preventionTips":["Pin optional extras (huggingface, tiktoken) in the deployment manifest/Dockerfile so they survive rebuilds.","In tests/CI without network, prefer 'estimator' mode so availability never depends on the Hub."],"tags":["dependencies","tokenizer","huggingface","import"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}