{"record":{"id":"41709eddb25f97e9","repo":"zylon-ai/private-gpt","slug":"tokenizer-mode-tokenizer-mode-not-found","errorCode":null,"errorMessage":"Tokenizer mode {tokenizer_mode} not found.","messagePattern":"Tokenizer mode (.+?) not found\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/registry.py","lineNumber":109,"sourceCode":"}\n\n\nclass TokenizerRegistry:\n    @staticmethod\n    @auto_discover_model(\n        enabled=True,\n        tokenizer_only=True,\n        raise_on_error=False,\n    )\n    def get_tokenizer(\n        tokenizer_mode: str,\n        **kwargs: Any,\n    ) -> TokenizerBase:\n        factory = _EXTERNAL_TOKENIZER_FACTORIES.get(\n            tokenizer_mode\n        ) or _BUILTIN_TOKENIZER_FACTORIES.get(tokenizer_mode)\n        if factory is None:\n            raise ValueError(f\"Tokenizer mode {tokenizer_mode} not found.\")\n        return factory(**kwargs)\n","sourceCodeStart":91,"sourceCodeEnd":111,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/registry.py#L91-L111","documentation":"Raised by get_tokenizer when the requested tokenizer_mode matches neither the plugin-provided _EXTERNAL_TOKENIZER_FACTORIES (populated via enabled entry points, tokenizer_only=True, raise_on_error=False) nor the built-in _BUILTIN_TOKENIZER_FACTORIES. It is a straightforward registry-miss ValueError: the string you passed is not a registered tokenizer mode.","triggerScenarios":"Calling get_tokenizer(tokenizer_mode, ...) with a typo'd or unregistered mode string, e.g. 'remote' instead of 'remote_tokenize', 'hf' instead of 'huggingface'; or referencing a mode provided by a plugin package that is not installed or whose entry point failed to load (raise_on_error=False means broken plugins are silently skipped).","commonSituations":"Typos in settings (llm.tokenizer_mode=hugging_face vs huggingface); upgrading private-gpt where a mode was renamed; a third-party tokenizer plugin package not installed in the environment, so its entry point never registers.","solutions":["Use a registered mode name — check the built-ins (e.g. 'estimator', 'remote_tokenize', 'huggingface', 'tiktoken') and any plugin-provided names.","Print/inspect the registered factories (keys of _EXTERNAL_TOKENIZER_FACTORIES and _BUILTIN_TOKENIZER_FACTORIES) to see exactly which modes exist in your environment.","If the mode should come from a plugin, install the plugin package and verify its entry point is discovered (a broken entry point is skipped silently because raise_on_error=False).","Omit tokenizer_mode to let the default tokenizer chain pick a working implementation."],"exampleFix":"# before\ntok = get_tokenizer('remote')  # ValueError: mode not found\n\n# after\nfrom private_gpt.components.llm.tokenizers.registry import _BUILTIN_TOKENIZER_FACTORIES, _EXTERNAL_TOKENIZER_FACTORIES\nprint(list(_EXTERNAL_TOKENIZER_FACTORIES), list(_BUILTIN_TOKENIZER_FACTORIES))\ntok = get_tokenizer('remote_tokenize')","handlingStrategy":"validation","validationCode":"from private_gpt.components.llm.tokenizers.registry import (_BUILTIN_TOKENIZER_FACTORIES,\n    _EXTERNAL_TOKENIZER_FACTORIES)\n\nvalid_modes = set(_EXTERNAL_TOKENIZER_FACTORIES) | set(_BUILTIN_TOKENIZER_FACTORIES)\nassert tokenizer_mode in valid_modes, f'use one of {sorted(valid_modes)}'","typeGuard":"def is_known_tokenizer_mode(mode: str) -> bool:\n    return mode in _EXTERNAL_TOKENIZER_FACTORIES or mode in _BUILTIN_TOKENIZER_FACTORIES","tryCatchPattern":"try:\n    tok = get_tokenizer(mode, **kwargs)\nexcept ValueError as e:\n    if 'not found' in str(e):\n        log_available_modes(); raise","preventionTips":["Validate tokenizer_mode against the registry keys at config-load time, not at first use.","Treat a silently-skipped plugin entry point (raise_on_error=False) as a red flag: log registered modes at boot."],"tags":["registry","tokenizer","configuration","typo"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}