{"record":{"id":"33022a952e797eb9","repo":"zylon-ai/private-gpt","slug":"transformers-dependencies-are-not-installed","errorCode":null,"errorMessage":"Transformers dependencies are not installed.","messagePattern":"Transformers dependencies are not installed\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/huggingface.py","lineNumber":69,"sourceCode":"        **kwargs: Any,\n    ) -> \"HuggingFaceTokenizer\":\n        \"\"\"Load tokenizer from pretrained model with intelligent caching.\n\n        If the model is already cached locally, it will automatically use\n        offline mode to avoid network calls.\n\n        Args:\n            model_id: Model identifier or local path\n            local_files_only: Force offline mode (no downloads)\n            cache_dir: Custom cache directory\n            force_download: Force re-download even if cached\n            trust_remote_code: Allow custom code from model repositories\n            **kwargs: Additional arguments for AutoProcessor\n        \"\"\"\n        try:\n            from transformers import AutoProcessor  # ty:ignore[unresolved-import]\n        except ImportError as e:\n            raise ImportError(\n                format_missing_dependency_message(\n                    \"Transformers\",\n                )\n            ) from e\n\n        try:\n            is_multimodal = False\n            processor = None\n            loaded: Any = AutoProcessor.from_pretrained(\n                pretrained_model_name_or_path=model_id,\n                local_files_only=local_files_only,\n                cache_dir=cache_dir,\n                force_download=force_download,\n                trust_remote_code=trust_remote_code,\n                **kwargs,\n            )\n\n            # Extract tokenizer from multimodal processor if needed","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/huggingface.py#L51-L87","documentation":"HuggingFaceTokenizer.from_pretrained imports transformers.AutoProcessor inside the method so that the rest of the module works without transformers installed. If the import fails it raises ImportError with a formatted missing-dependency message for 'Transformers'.","triggerScenarios":"Calling HuggingFaceTokenizer.from_pretrained (directly or via TokenizerRegistry modes 'huggingface'/'chat') in an environment lacking the transformers package.","commonSituations":"Slack/minimal installs that skipped the tokenizer extras; Docker images trimmed for remote-API-only deployments (no local models) where tokenizer_mode was later changed to 'huggingface'; CI environments with a reduced dependency set.","solutions":["Install transformers (or the project extra that pulls it in, e.g. the local-LLM extra): uv sync --inexact --extra <local-llm-extra>.","Verify: python -c \"from transformers import AutoProcessor\".","If you only use remote APIs and don't need real tokenization, switch tokenizer_mode to 'estimator' or 'remote'."],"exampleFix":"# before: tokenizer_mode='huggingface' without transformers installed\n# after\nuv sync --inexact --extra llm-huggingface  # or pip install transformers","handlingStrategy":"validation","validationCode":"def transformers_available() -> bool:\n    try:\n        from transformers import AutoProcessor  # noqa: F401\n        return True\n    except ImportError:\n        return False","typeGuard":null,"tryCatchPattern":"try:\n    tok = HuggingFaceTokenizer.from_pretrained(model_id)\nexcept ImportError as e:\n    logger.warning('transformers missing; falling back to estimator tokenizer')\n    tok = TokenizerRegistry.get_tokenizer('estimator', model_id=model_id)","preventionTips":["Include the tokenizer extras in deployments that set tokenizer_mode to 'huggingface' or 'chat'.","Add a startup dependency probe when local model support is enabled.","Document which config flags require which extras."],"tags":["dependencies","import","huggingface","tokenizer"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}