BerriAI/litellm · error · ValueError

Unsupported tokenizer type

Error message

Unsupported tokenizer type

What it means

_get_count_function resolves a tokenizer spec (from custom_tokenizer= or auto-selected for the model) and only supports two types: 'huggingface_tokenizer' and 'openai_tokenizer'. A spec whose 'type' is anything else raises this - usually a malformed custom_tokenizer dict, or _select_tokenizer returning an unexpected entry for the model name.

Source

Thrown at litellm/litellm_core_utils/token_counter.py:549

                enc: Final = tokenizer_json["tokenizer"].encode(text)
                return len(enc.ids)

        elif tokenizer_json["type"] == "openai_tokenizer":
            model_to_use: Final = _fix_model_name(model)
            try:
                if "gpt-4o" in model_to_use:
                    encoding = tiktoken.get_encoding("o200k_base")
                else:
                    encoding = tiktoken.encoding_for_model(model_to_use)
            except KeyError:
                print_verbose("Warning: model not found. Using cl100k_base encoding.")
                encoding = tiktoken.get_encoding("cl100k_base")

            def count_tokens(text: str) -> int:
                return len(encoding.encode(text, disallowed_special=()))

        else:
            raise ValueError("Unsupported tokenizer type")
    else:

        def count_tokens(text: str) -> int:
            return len(default_encoding.encode(text, disallowed_special=()))

    return count_tokens


def _fix_model_name(model: str) -> str:
    """We normalize some model names to others"""
    if model in litellm.azure_llms:
        # azure llms use gpt-35-turbo instead of gpt-3.5-turbo 🙃
        return model.replace("-35", "-3.5")
    elif model in litellm.open_ai_chat_completion_models:
        return model
    else:
        return "gpt-3.5-turbo"

View on GitHub (pinned to 6c2dcb801b)

Solutions

1. Check litellm.utils._select_tokenizer(model)['type'] for your model; if it is neither supported value, pass custom_tokenizer explicitly with type='openai_tokenizer' (tiktoken encoding) or type='huggingface_tokenizer'.
2. When building a custom_tokenizer dict, use exactly {"type": "huggingface_tokenizer", "tokenizer": <HF tokenizer>} or {"type": "openai_tokenizer", "tokenizer": <tiktoken encoding>} per the docs.
3. Update litellm - model-to-tokenizer mapping fixes land regularly.

Example fix

# before
litellm.token_counter(model="my-finetuned-model",
    custom_tokenizer={"type": "tiktoken", "tokenizer": "cl100k_base"})

# after
import tiktoken
litellm.token_counter(model="my-finetuned-model",
    custom_tokenizer={"type": "openai_tokenizer", "tokenizer": tiktoken.get_encoding("cl100k_base")})

from litellm.utils import _select_tokenizer
spec = _select_tokenizer(model)
assert spec["type"] in ("huggingface_tokenizer", "openai_tokenizer"), spec["type"]
n = litellm.token_counter(model=model, messages=msgs)

def tokenizer_type_supported(model: str) -> bool:
    try:
        return _select_tokenizer(model)["type"] in ("huggingface_tokenizer", "openai_tokenizer")
    except Exception:
        return False

When it happens

Trigger: Passing custom_tokenizer with a 'type' other than the two supported values (e.g. type='tiktoken'); calling token_counter(model=<unrecognized model>) where litellm's _select_tokenizer returns a type this function does not handle; a custom tokenizer dict with wrong 'type' semantics.

Common situations: Users wiring their own tokenizer per the docs but guessing the type string; new or renamed models hitting an unmapped branch in _select_tokenizer; forked litellm versions where the spec vocabulary drifted.

Related errors

• Error: {response.status_code} - {response.text}
• Missing Authorization header
• Invalid bearer token
• Invalid API key
• Prompt '{prompt_id}' not found

AI-assisted analysis of BerriAI/litellm@6c2dcb801b (2026-08-15). Data as JSON: /api/errors/a18549cdb9721b7b. Report an issue: GitHub.