{"record":{"id":"879e98864a4c3f92","repo":"zylon-ai/private-gpt","slug":"tiktoken-dependencies-are-not-installed-install-w","errorCode":null,"errorMessage":"tiktoken dependencies are not installed. Install with `uv sync --inexact --extra tiktoken`.","messagePattern":"tiktoken dependencies are not installed\\. Install with `uv sync --inexact --extra tiktoken`\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/tiktoken.py","lineNumber":32,"sourceCode":"\n\nclass TikTokenTokenizer(TokenizerBase):\n    \"\"\"Tokenizer backed by tiktoken for local token counting.\"\"\"\n\n    def __init__(self, encoding: Any, encoding_name: str) -> None:\n        self._encoding = encoding\n        self._encoding_name = encoding_name\n\n    @classmethod\n    def from_pretrained(\n        cls,\n        model_id: str,\n        **kwargs: Any,\n    ) -> \"TikTokenTokenizer\":\n        try:\n            import tiktoken\n        except ImportError as e:\n            raise ImportError(format_missing_dependency_message(\"tiktoken\")) from e\n\n        # An explicit encoding_name (e.g. \"cl100k_base\") takes precedence over\n        # auto-detection so callers using OpenAI-compatible/local model names\n        # can always specify the exact encoding they need.\n        explicit_encoding_name: str | None = kwargs.get(\"encoding_name\")\n        if explicit_encoding_name:\n            encoding = tiktoken.get_encoding(explicit_encoding_name)\n            return cls(encoding=encoding, encoding_name=explicit_encoding_name)\n\n        if model_id:\n            # 1. Try the tiktoken model registry (covers all known OpenAI model ids).\n            try:\n                encoding = tiktoken.encoding_for_model(model_id)\n                return cls(encoding=encoding, encoding_name=encoding.name)\n            except (KeyError, ValueError):\n                pass\n\n            # 2. model_id might itself be an encoding name (e.g. \"cl100k_base\").","sourceCodeStart":14,"sourceCodeEnd":50,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/tiktoken.py#L14-L50","documentation":"Raised by TikTokenTokenizer.from_pretrained when `import tiktoken` fails. The optional tiktoken dependency is behind an extra, and the error message states the exact install command: uv sync --inexact --extra tiktoken. This fires only when the tiktoken tokenizer mode (or the default chain reaching it) is requested in an environment lacking the package; it is purely an environment/dependency issue, not a model or network one.","triggerScenarios":"get_tokenizer('tiktoken', ...) or TikTokenTokenizer.from_pretrained(model_id, ...) in an environment where tiktoken is not installed; the default tokenizer chain falling through to tiktoken after HF failed; a fresh deploy created with a minimal dependency set.","commonSituations":"Installing private-gpt without the tiktoken extra; Docker images trimmed of optional extras; lockfile refresh dropping the extra after a config change; switching tokenizer_mode to 'tiktoken' on an existing slim install.","solutions":["Install the extra exactly as the message says: uv sync --inexact --extra tiktoken (or pip install tiktoken).","If you did not intend tiktoken, switch tokenizer_mode to 'estimator' or another extra-free mode.","Add the extra to your deployment manifest/Dockerfile so rebuilds keep it.","Re-run after install in the same virtualenv the app uses (verify with `python -c \"import tiktoken\"`)."],"exampleFix":"# before\nTikTokenTokenizer.from_pretrained('gpt-4o')  # ImportError\n\n# after (shell)\n# uv sync --inexact --extra tiktoken\nTikTokenTokenizer.from_pretrained('gpt-4o')","handlingStrategy":"try-catch","validationCode":"try:\n    import tiktoken  # noqa: F401\n    HAS_TIKTOKEN = True\nexcept ImportError:\n    HAS_TIKTOKEN = False\n\nmode = 'tiktoken' if HAS_TIKTOKEN else 'estimator'","typeGuard":null,"tryCatchPattern":"try:\n    tok = get_tokenizer('tiktoken', **kwargs)\nexcept ImportError:\n    tok = get_tokenizer('estimator', **kwargs)","preventionTips":["Declare the tiktoken extra in the same commit that switches tokenizer_mode to tiktoken.","Add a boot-time `import tiktoken` smoke check when the mode is configured."],"tags":["dependencies","tokenizer","tiktoken","import"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}