What causes "Tokenizer mode {tokenizer_mode} not found." in zylon-ai/private-gpt?

Use a registered mode name — check the built-ins (e.g. 'estimator', 'remote_tokenize', 'huggingface', 'tiktoken') and any plugin-provided names.

zylon-ai/private-gpt · error · ValueError

Tokenizer mode {tokenizer_mode} not found.

Error message

Tokenizer mode {tokenizer_mode} not found.

What it means

Raised by get_tokenizer when the requested tokenizer_mode matches neither the plugin-provided _EXTERNAL_TOKENIZER_FACTORIES (populated via enabled entry points, tokenizer_only=True, raise_on_error=False) nor the built-in _BUILTIN_TOKENIZER_FACTORIES. It is a straightforward registry-miss ValueError: the string you passed is not a registered tokenizer mode.

Source

Thrown at private_gpt/components/llm/tokenizers/registry.py:109

}


class TokenizerRegistry:
    @staticmethod
    @auto_discover_model(
        enabled=True,
        tokenizer_only=True,
        raise_on_error=False,
    )
    def get_tokenizer(
        tokenizer_mode: str,
        **kwargs: Any,
    ) -> TokenizerBase:
        factory = _EXTERNAL_TOKENIZER_FACTORIES.get(
            tokenizer_mode
        ) or _BUILTIN_TOKENIZER_FACTORIES.get(tokenizer_mode)
        if factory is None:
            raise ValueError(f"Tokenizer mode {tokenizer_mode} not found.")
        return factory(**kwargs)

View on GitHub (pinned to 4a030776a3)

Solutions

Use a registered mode name — check the built-ins (e.g. 'estimator', 'remote_tokenize', 'huggingface', 'tiktoken') and any plugin-provided names.
Print/inspect the registered factories (keys of _EXTERNAL_TOKENIZER_FACTORIES and _BUILTIN_TOKENIZER_FACTORIES) to see exactly which modes exist in your environment.
If the mode should come from a plugin, install the plugin package and verify its entry point is discovered (a broken entry point is skipped silently because raise_on_error=False).
Omit tokenizer_mode to let the default tokenizer chain pick a working implementation.

Example fix

# before
tok = get_tokenizer('remote')  # ValueError: mode not found

# after
from private_gpt.components.llm.tokenizers.registry import _BUILTIN_TOKENIZER_FACTORIES, _EXTERNAL_TOKENIZER_FACTORIES
print(list(_EXTERNAL_TOKENIZER_FACTORIES), list(_BUILTIN_TOKENIZER_FACTORIES))
tok = get_tokenizer('remote_tokenize')

Defensive patterns

Strategy: validation

Validate before calling

from private_gpt.components.llm.tokenizers.registry import (_BUILTIN_TOKENIZER_FACTORIES,
    _EXTERNAL_TOKENIZER_FACTORIES)

valid_modes = set(_EXTERNAL_TOKENIZER_FACTORIES) | set(_BUILTIN_TOKENIZER_FACTORIES)
assert tokenizer_mode in valid_modes, f'use one of {sorted(valid_modes)}'

Type guard

def is_known_tokenizer_mode(mode: str) -> bool:
    return mode in _EXTERNAL_TOKENIZER_FACTORIES or mode in _BUILTIN_TOKENIZER_FACTORIES

Try / catch

try:
    tok = get_tokenizer(mode, **kwargs)
except ValueError as e:
    if 'not found' in str(e):
        log_available_modes(); raise

Prevention

Validate tokenizer_mode against the registry keys at config-load time, not at first use.
Treat a silently-skipped plugin entry point (raise_on_error=False) as a red flag: log registered modes at boot.

When it happens

Trigger: Calling get_tokenizer(tokenizer_mode, ...) with a typo'd or unregistered mode string, e.g. 'remote' instead of 'remote_tokenize', 'hf' instead of 'huggingface'; or referencing a mode provided by a plugin package that is not installed or whose entry point failed to load (raise_on_error=False means broken plugins are silently skipped).

Common situations: Typos in settings (llm.tokenizer_mode=hugging_face vs huggingface); upgrading private-gpt where a mode was renamed; a third-party tokenizer plugin package not installed in the environment, so its entry point never registers.

Related errors

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/41709eddb25f97e9. Report an issue: GitHub.