{"record":{"id":"8a6a5c846676cdd9","repo":"microsoft/graphrag","slug":"encoding-name-must-be-specified-for-tiktoken-token","errorCode":null,"errorMessage":"encoding_name must be specified for TikToken tokenizer.","messagePattern":"encoding_name must be specified for TikToken tokenizer\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"packages/graphrag-llm/graphrag_llm/config/tokenizer_config.py","lineNumber":42,"sourceCode":"        description=\"The identifier for the tokenizer model. Example: openai/gpt-4o. Used by the litellm tokenizer.\",\n    )\n\n    encoding_name: str | None = Field(\n        default=None,\n        description=\"The encoding name for the tokenizer. Example: gpt-4o.\",\n    )\n\n    def _validate_litellm_config(self) -> None:\n        \"\"\"Validate LiteLLM tokenizer configuration.\"\"\"\n        if self.model_id is None or self.model_id.strip() == \"\":\n            msg = \"model_id must be specified for LiteLLM tokenizer.\"\n            raise ValueError(msg)\n\n    def _validate_tiktoken_config(self) -> None:\n        \"\"\"Validate TikToken tokenizer configuration.\"\"\"\n        if self.encoding_name is None or self.encoding_name.strip() == \"\":\n            msg = \"encoding_name must be specified for TikToken tokenizer.\"\n            raise ValueError(msg)\n\n    @model_validator(mode=\"after\")\n    def _validate_model(self):\n        \"\"\"Validate the tokenizer configuration based on its type.\"\"\"\n        if self.type == TokenizerType.LiteLLM:\n            self._validate_litellm_config()\n        elif self.type == TokenizerType.Tiktoken:\n            self._validate_tiktoken_config()\n        return self\n","sourceCodeStart":24,"sourceCodeEnd":52,"githubUrl":"https://github.com/microsoft/graphrag/blob/f40e9a26ce62ba0b3fef8837d24aafdcc6e6c704/packages/graphrag-llm/graphrag_llm/config/tokenizer_config.py#L24-L52","documentation":"TokenizerConfig validation requires encoding_name when using the TikToken tokenizer, since tiktoken.get_encode needs a named encoding (e.g. cl100k_base). Missing or blank values are rejected.","triggerScenarios":"TokenizerConfig(type=TokenizerType.TikToken) with encoding_name=None or \"\" in settings.yaml or code.","commonSituations":"Switching tokenizer type to tiktoken without adding encoding_name; expecting the encoding to default from the model name; blank value from templated YAML.","solutions":["Set encoding_name to a known tiktoken encoding, e.g. \"cl100k_base\" or \"o200k_base\"","If you wanted automatic model-based encoding, use the LiteLLM tokenizer type with model_id instead"],"exampleFix":"# before\nTokenizerConfig(type=\"tiktoken\")\n# after\nTokenizerConfig(type=\"tiktoken\", encoding_name=\"cl100k_base\")","handlingStrategy":"validation","validationCode":"if tok_cfg.get(\"type\") == \"tiktoken\" and not (tok_cfg.get(\"encoding_name\") or \"\").strip():\n    tok_cfg[\"encoding_name\"] = \"cl100k_base\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin a known encoding (cl100k_base/o200k_base) in settings templates","Add a config linter that fills tokenizer defaults before validation"],"tags":["tokenizer","tiktoken","config-validation","required-field"],"backgroundTag":"missing-required-config-field","analyzedSha":"f40e9a26ce62ba0b3fef8837d24aafdcc6e6c704","analyzedAt":"2026-08-27T11:16:29.677Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}