zylon-ai/private-gpt · error · ImportError

HuggingFaceTokenizer is not available with the given configu

Error message

HuggingFaceTokenizer is not available with the given configuration.

What it means

Raised by _build_huggingface_tokenizer when HuggingFaceTokenizer.is_available(**kwargs) returns False. It is an ImportError, signalling that the local HuggingFace tokenizer path cannot be used — most often because the 'transformers' dependency (huggingface extra) is not installed, or the required model/config cannot be resolved. Note that the default-tokenizer chain in the same file catches ImportError (and broadly Exception) and falls through to the next builder, so users normally only see this when explicitly requesting the huggingface mode.

Source

Thrown at private_gpt/components/llm/tokenizers/registry.py:59

    return EstimatorTokenizer.from_pretrained(**kwargs)


def _build_remote_tokenizer(**kwargs: Any) -> TokenizerBase:
    from private_gpt.components.llm.tokenizers.remote import RemoteTokenizeTokenizer

    if not RemoteTokenizeTokenizer.is_available(**kwargs):
        raise ValueError(
            "RemoteTokenizeTokenizer is not available with the given configuration."
        )

    return RemoteTokenizeTokenizer.from_pretrained(**kwargs)


def _build_huggingface_tokenizer(**kwargs: Any) -> TokenizerBase:
    from private_gpt.components.llm.tokenizers.huggingface import HuggingFaceTokenizer

    if not HuggingFaceTokenizer.is_available(**kwargs):
        raise ImportError(
            "HuggingFaceTokenizer is not available with the given configuration."
        )

    return HuggingFaceTokenizer.from_pretrained(**kwargs)


def _build_default_tokenizer(**kwargs: Any) -> TokenizerBase:
    # 1. HF as initial tokenizer
    try:
        return _build_huggingface_tokenizer(**kwargs)
    except (ImportError, Exception):
        pass

    # 2. Use the remote tokenizer
    try:
        return _build_remote_tokenizer(**kwargs)
    except (ImportError, ValueError, Exception):
        pass

View on GitHub (pinned to 4a030776a3)

Solutions

Install the huggingface extra: uv sync --inexact --extra huggingface (or pip install transformers tokenizers).
If the model is gated/private, set a valid HF_TOKEN and confirm the model_id is accessible.
Check the fallback chain: if you do not specifically need HF, use 'estimator' or 'tiktoken' mode which need no Hub access.
If is_available still fails, run HuggingFaceTokenizer.is_available(**kwargs) directly to see which dependency/config check rejects it.

Example fix

# before
tokenizer = get_tokenizer('huggingface')  # ImportError

# after (shell)
# uv sync --inexact --extra huggingface
tokenizer = get_tokenizer('huggingface')

Defensive patterns

Strategy: validation

Validate before calling

from private_gpt.components.llm.tokenizers.huggingface import HuggingFaceTokenizer

if not HuggingFaceTokenizer.is_available(**kwargs):
    raise SystemExit('Install the huggingface extra or switch tokenizer_mode')

Try / catch

try:
    tok = get_tokenizer('huggingface', **kwargs)
except ImportError:
    tok = get_tokenizer('estimator', **kwargs)

Prevention

Pin optional extras (huggingface, tiktoken) in the deployment manifest/Dockerfile so they survive rebuilds.
In tests/CI without network, prefer 'estimator' mode so availability never depends on the Hub.

When it happens

Trigger: Explicitly requesting tokenizer mode 'huggingface' via get_tokenizer('huggingface', ...) when the huggingface extra is absent, or when is_available fails for the given model_id/config (missing transformers/tokenizers packages, unreachable Hub, gated model without credentials).

Common situations: Slim installation without the huggingface extra (uv sync --inexact --extra huggingface missing); CI environments with no network access to huggingface.co; gated/private models without a valid HF token; dependency upgrades dropping transformers.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/7e2e0fff82db4579. Report an issue: GitHub.