microsoft/graphrag · error · ValueError

encoding_name must be specified for TikToken tokenizer.

Error message

encoding_name must be specified for TikToken tokenizer.

What it means

TokenizerConfig validation requires encoding_name when using the TikToken tokenizer, since tiktoken.get_encode needs a named encoding (e.g. cl100k_base). Missing or blank values are rejected.

Source

Thrown at packages/graphrag-llm/graphrag_llm/config/tokenizer_config.py:42

        description="The identifier for the tokenizer model. Example: openai/gpt-4o. Used by the litellm tokenizer.",
    )

    encoding_name: str | None = Field(
        default=None,
        description="The encoding name for the tokenizer. Example: gpt-4o.",
    )

    def _validate_litellm_config(self) -> None:
        """Validate LiteLLM tokenizer configuration."""
        if self.model_id is None or self.model_id.strip() == "":
            msg = "model_id must be specified for LiteLLM tokenizer."
            raise ValueError(msg)

    def _validate_tiktoken_config(self) -> None:
        """Validate TikToken tokenizer configuration."""
        if self.encoding_name is None or self.encoding_name.strip() == "":
            msg = "encoding_name must be specified for TikToken tokenizer."
            raise ValueError(msg)

    @model_validator(mode="after")
    def _validate_model(self):
        """Validate the tokenizer configuration based on its type."""
        if self.type == TokenizerType.LiteLLM:
            self._validate_litellm_config()
        elif self.type == TokenizerType.Tiktoken:
            self._validate_tiktoken_config()
        return self

View on GitHub (pinned to f40e9a26ce)

Solutions

1. Set encoding_name to a known tiktoken encoding, e.g. "cl100k_base" or "o200k_base"
2. If you wanted automatic model-based encoding, use the LiteLLM tokenizer type with model_id instead

Example fix

# before
TokenizerConfig(type="tiktoken")
# after
TokenizerConfig(type="tiktoken", encoding_name="cl100k_base")

if tok_cfg.get("type") == "tiktoken" and not (tok_cfg.get("encoding_name") or "").strip():
    tok_cfg["encoding_name"] = "cl100k_base"

When it happens

Trigger: TokenizerConfig(type=TokenizerType.TikToken) with encoding_name=None or "" in settings.yaml or code.

Common situations: Switching tokenizer type to tiktoken without adding encoding_name; expecting the encoding to default from the model name; blank value from templated YAML.

Related errors

• model_id must be specified for LiteLLM tokenizer.
• At least one of requests_per_period or tokens_per_period mus
• api_base must be specified with the 'azure' model provider.
• azure_deployment_name should not be specified for non-Azure
• api_key should not be set when using Azure Managed Identity.

AI-assisted analysis of microsoft/graphrag@f40e9a26ce (2026-08-27). Data as JSON: /api/errors/8a6a5c846676cdd9. Report an issue: GitHub.