{"record":{"id":"a18549cdb9721b7b","repo":"BerriAI/litellm","slug":"unsupported-tokenizer-type","errorCode":null,"errorMessage":"Unsupported tokenizer type","messagePattern":"Unsupported tokenizer type","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"litellm/litellm_core_utils/token_counter.py","lineNumber":549,"sourceCode":"                enc: Final = tokenizer_json[\"tokenizer\"].encode(text)\n                return len(enc.ids)\n\n        elif tokenizer_json[\"type\"] == \"openai_tokenizer\":\n            model_to_use: Final = _fix_model_name(model)\n            try:\n                if \"gpt-4o\" in model_to_use:\n                    encoding = tiktoken.get_encoding(\"o200k_base\")\n                else:\n                    encoding = tiktoken.encoding_for_model(model_to_use)\n            except KeyError:\n                print_verbose(\"Warning: model not found. Using cl100k_base encoding.\")\n                encoding = tiktoken.get_encoding(\"cl100k_base\")\n\n            def count_tokens(text: str) -> int:\n                return len(encoding.encode(text, disallowed_special=()))\n\n        else:\n            raise ValueError(\"Unsupported tokenizer type\")\n    else:\n\n        def count_tokens(text: str) -> int:\n            return len(default_encoding.encode(text, disallowed_special=()))\n\n    return count_tokens\n\n\ndef _fix_model_name(model: str) -> str:\n    \"\"\"We normalize some model names to others\"\"\"\n    if model in litellm.azure_llms:\n        # azure llms use gpt-35-turbo instead of gpt-3.5-turbo 🙃\n        return model.replace(\"-35\", \"-3.5\")\n    elif model in litellm.open_ai_chat_completion_models:\n        return model\n    else:\n        return \"gpt-3.5-turbo\"\n","sourceCodeStart":531,"sourceCodeEnd":567,"githubUrl":"https://github.com/BerriAI/litellm/blob/6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d/litellm/litellm_core_utils/token_counter.py#L531-L567","documentation":"_get_count_function resolves a tokenizer spec (from custom_tokenizer= or auto-selected for the model) and only supports two types: 'huggingface_tokenizer' and 'openai_tokenizer'. A spec whose 'type' is anything else raises this - usually a malformed custom_tokenizer dict, or _select_tokenizer returning an unexpected entry for the model name.","triggerScenarios":"Passing custom_tokenizer with a 'type' other than the two supported values (e.g. type='tiktoken'); calling token_counter(model=<unrecognized model>) where litellm's _select_tokenizer returns a type this function does not handle; a custom tokenizer dict with wrong 'type' semantics.","commonSituations":"Users wiring their own tokenizer per the docs but guessing the type string; new or renamed models hitting an unmapped branch in _select_tokenizer; forked litellm versions where the spec vocabulary drifted.","solutions":["Check litellm.utils._select_tokenizer(model)['type'] for your model; if it is neither supported value, pass custom_tokenizer explicitly with type='openai_tokenizer' (tiktoken encoding) or type='huggingface_tokenizer'.","When building a custom_tokenizer dict, use exactly {\"type\": \"huggingface_tokenizer\", \"tokenizer\": <HF tokenizer>} or {\"type\": \"openai_tokenizer\", \"tokenizer\": <tiktoken encoding>} per the docs.","Update litellm - model-to-tokenizer mapping fixes land regularly."],"exampleFix":"# before\nlitellm.token_counter(model=\"my-finetuned-model\",\n    custom_tokenizer={\"type\": \"tiktoken\", \"tokenizer\": \"cl100k_base\"})\n\n# after\nimport tiktoken\nlitellm.token_counter(model=\"my-finetuned-model\",\n    custom_tokenizer={\"type\": \"openai_tokenizer\", \"tokenizer\": tiktoken.get_encoding(\"cl100k_base\")})","handlingStrategy":"validation","validationCode":"from litellm.utils import _select_tokenizer\nspec = _select_tokenizer(model)\nassert spec[\"type\"] in (\"huggingface_tokenizer\", \"openai_tokenizer\"), spec[\"type\"]\nn = litellm.token_counter(model=model, messages=msgs)","typeGuard":"def tokenizer_type_supported(model: str) -> bool:\n    try:\n        return _select_tokenizer(model)[\"type\"] in (\"huggingface_tokenizer\", \"openai_tokenizer\")\n    except Exception:\n        return False","tryCatchPattern":null,"preventionTips":["For custom models always pass a fully-formed custom_tokenizer dict with a supported 'type'.","Pin and periodically upgrade litellm so new models map to known tokenizers."],"tags":["token-counter","tokenizer","configuration"],"backgroundTag":null,"analyzedSha":"6c2dcb801bf2b75c18f1bb24140e7cf57465cc4d","analyzedAt":"2026-08-15T07:12:03.035Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}