zylon-ai/private-gpt · error · ImportError

tiktoken dependencies are not installed. Install with `uv sy

Error message

tiktoken dependencies are not installed. Install with `uv sync --inexact --extra tiktoken`.

What it means

Raised by TikTokenTokenizer.from_pretrained when `import tiktoken` fails. The optional tiktoken dependency is behind an extra, and the error message states the exact install command: uv sync --inexact --extra tiktoken. This fires only when the tiktoken tokenizer mode (or the default chain reaching it) is requested in an environment lacking the package; it is purely an environment/dependency issue, not a model or network one.

Source

Thrown at private_gpt/components/llm/tokenizers/tiktoken.py:32


class TikTokenTokenizer(TokenizerBase):
    """Tokenizer backed by tiktoken for local token counting."""

    def __init__(self, encoding: Any, encoding_name: str) -> None:
        self._encoding = encoding
        self._encoding_name = encoding_name

    @classmethod
    def from_pretrained(
        cls,
        model_id: str,
        **kwargs: Any,
    ) -> "TikTokenTokenizer":
        try:
            import tiktoken
        except ImportError as e:
            raise ImportError(format_missing_dependency_message("tiktoken")) from e

        # An explicit encoding_name (e.g. "cl100k_base") takes precedence over
        # auto-detection so callers using OpenAI-compatible/local model names
        # can always specify the exact encoding they need.
        explicit_encoding_name: str | None = kwargs.get("encoding_name")
        if explicit_encoding_name:
            encoding = tiktoken.get_encoding(explicit_encoding_name)
            return cls(encoding=encoding, encoding_name=explicit_encoding_name)

        if model_id:
            # 1. Try the tiktoken model registry (covers all known OpenAI model ids).
            try:
                encoding = tiktoken.encoding_for_model(model_id)
                return cls(encoding=encoding, encoding_name=encoding.name)
            except (KeyError, ValueError):
                pass

            # 2. model_id might itself be an encoding name (e.g. "cl100k_base").

View on GitHub (pinned to 4a030776a3)

Solutions

Install the extra exactly as the message says: uv sync --inexact --extra tiktoken (or pip install tiktoken).
If you did not intend tiktoken, switch tokenizer_mode to 'estimator' or another extra-free mode.
Add the extra to your deployment manifest/Dockerfile so rebuilds keep it.
Re-run after install in the same virtualenv the app uses (verify with `python -c "import tiktoken"`).

Example fix

# before
TikTokenTokenizer.from_pretrained('gpt-4o')  # ImportError

# after (shell)
# uv sync --inexact --extra tiktoken
TikTokenTokenizer.from_pretrained('gpt-4o')

Defensive patterns

Strategy: try-catch

Validate before calling

try:
    import tiktoken  # noqa: F401
    HAS_TIKTOKEN = True
except ImportError:
    HAS_TIKTOKEN = False

mode = 'tiktoken' if HAS_TIKTOKEN else 'estimator'

Try / catch

try:
    tok = get_tokenizer('tiktoken', **kwargs)
except ImportError:
    tok = get_tokenizer('estimator', **kwargs)

Prevention

Declare the tiktoken extra in the same commit that switches tokenizer_mode to tiktoken.
Add a boot-time `import tiktoken` smoke check when the mode is configured.

When it happens

Trigger: get_tokenizer('tiktoken', ...) or TikTokenTokenizer.from_pretrained(model_id, ...) in an environment where tiktoken is not installed; the default tokenizer chain falling through to tiktoken after HF failed; a fresh deploy created with a minimal dependency set.

Common situations: Installing private-gpt without the tiktoken extra; Docker images trimmed of optional extras; lockfile refresh dropping the extra after a config change; switching tokenizer_mode to 'tiktoken' on an existing slim install.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/879e98864a4c3f92. Report an issue: GitHub.