{"record":{"id":"f06208d9f085e071","repo":"zylon-ai/private-gpt","slug":"remotetokenizetokenizer-is-not-available-with-the","errorCode":null,"errorMessage":"RemoteTokenizeTokenizer is not available with the given configuration.","messagePattern":"RemoteTokenizeTokenizer is not available with the given configuration\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/registry.py","lineNumber":48,"sourceCode":"\n\ndef _build_tiktoken_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.tiktoken import TikTokenTokenizer\n\n    return TikTokenTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_estimator_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.estimator import EstimatorTokenizer\n\n    return EstimatorTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_remote_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.remote import RemoteTokenizeTokenizer\n\n    if not RemoteTokenizeTokenizer.is_available(**kwargs):\n        raise ValueError(\n            \"RemoteTokenizeTokenizer is not available with the given configuration.\"\n        )\n\n    return RemoteTokenizeTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_huggingface_tokenizer(**kwargs: Any) -> TokenizerBase:\n    from private_gpt.components.llm.tokenizers.huggingface import HuggingFaceTokenizer\n\n    if not HuggingFaceTokenizer.is_available(**kwargs):\n        raise ImportError(\n            \"HuggingFaceTokenizer is not available with the given configuration.\"\n        )\n\n    return HuggingFaceTokenizer.from_pretrained(**kwargs)\n\n\ndef _build_default_tokenizer(**kwargs: Any) -> TokenizerBase:","sourceCodeStart":30,"sourceCodeEnd":66,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/registry.py#L30-L66","documentation":"Raised by the tokenizer registry's _build_remote_tokenizer factory when RemoteTokenizeTokenizer.is_available(**kwargs) returns False. The registry only constructs a remote tokenizer after an availability probe succeeds, so this error means the configuration passed to get_tokenizer (e.g. tokenizer mode 'remote_tokenize') is incomplete for remote operation — typically a missing or malformed endpoint URL or API key. It is a configuration-time ValueError thrown before any network call is made.","triggerScenarios":"Calling get_tokenizer('remote_tokenize', ...) (or any externally registered mode that maps to this factory) with kwargs where is_available() fails: no remote tokenizer base_url/endpoint configured, missing API key for a key-protected tokenizer service, or an unreachable/misformatted URL during the availability check.","commonSituations":"Deploying private-gpt with llm.tokenizer_mode=remote_tokenize in settings but forgetting the tokenizer server URL; rotating/removing the API key so the availability probe fails; pointing at a tokenizer service that is down at startup.","solutions":["Set the remote tokenizer endpoint (and API key if required) in your settings/profile so RemoteTokenizeTokenizer.is_available(**kwargs) returns True.","Call RemoteTokenizeTokenizer.is_available(**kwargs) directly before get_tokenizer to confirm which config field is failing.","If no remote service is intended, switch tokenizer_mode to a local mode ('huggingface', 'tiktoken', 'estimator') or omit it to use the default chain.","Verify the remote tokenizer service is running and reachable from the app host (URL scheme, port, TLS)."],"exampleFix":"# before\ntokenizer = get_tokenizer('remote_tokenize')  # ValueError: not available\n\n# after\nsettings.tokenizer_remote.url = 'http://localhost:8000'\nassert RemoteTokenizeTokenizer.is_available()\ntokenizer = get_tokenizer('remote_tokenize')","handlingStrategy":"validation","validationCode":"from private_gpt.components.llm.tokenizers.remote import RemoteTokenizeTokenizer\n\nok = RemoteTokenizeTokenizer.is_available(**tokenizer_kwargs)\nif not ok:\n    # fix config (url/api key) before requesting the tokenizer\n    ...","typeGuard":null,"tryCatchPattern":"try:\n    tok = get_tokenizer('remote_tokenize', **kwargs)\nexcept ValueError as e:\n    if 'not available' in str(e):\n        tok = get_tokenizer('estimator')  # deliberate fallback mode\n    else:\n        raise","preventionTips":["Run is_available(**kwargs) in a startup health check and fail deployment loudly when the remote tokenizer is configured but unreachable.","Keep the remote tokenizer URL/key in one settings source and validate them at load time (non-empty, valid URL scheme)."],"tags":["configuration","tokenizer","validation","startup"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}