microsoft/graphrag · error · ValueError

TokenizerConfig.type '{strategy}' is not registered in the T

Error message

TokenizerConfig.type '{strategy}' is not registered in the TokenizerFactory. Registered strategies: {', '.join(tokenizer_factory.keys())}

What it means

The tokenizer factory registers a fixed set of strategies (e.g. Tiktoken) in its match statement. Setting TokenizerConfig.type to any unregistered name hits the wildcard case and raises ValueError listing valid strategies.

Source

Thrown at packages/graphrag-llm/graphrag_llm/tokenizer/tokenizer_factory.py:84

                register_tokenizer(
                    TokenizerType.LiteLLM,
                    LiteLLMTokenizer,
                    scope="singleton",
                )
            case TokenizerType.Tiktoken:
                from graphrag_llm.tokenizer.tiktoken_tokenizer import (
                    TiktokenTokenizer,
                )

                register_tokenizer(
                    TokenizerType.Tiktoken,
                    TiktokenTokenizer,
                    scope="singleton",
                )
            case _:
                msg = f"TokenizerConfig.type '{strategy}' is not registered in the TokenizerFactory. Registered strategies: {', '.join(tokenizer_factory.keys())}"
                raise ValueError(msg)

    return tokenizer_factory.create(
        strategy=strategy,
        init_args=init_args,
    )

View on GitHub (pinned to f40e9a26ce)

Solutions

Set TokenizerConfig.type to one of the strategies printed in the error message
Verify exact casing/whitespace in the config source
Register the custom tokenizer in TokenizerFactory if you genuinely need a new backend
Align the graphrag-llm version with the config schema you followed

Example fix

# before
cfg = TokenizerConfig(type='hf')
tok = get_tokenizer(cfg)

# after
from graphrag_llm.tokenizer import TokenizerType
cfg = TokenizerConfig(type=TokenizerType.Tiktoken)
tok = get_tokenizer(cfg)

Defensive patterns

Strategy: validation

Validate before calling

from graphrag_llm.tokenizer.tokenizer_factory import tokenizer_factory
valid = set(tokenizer_factory.keys())
assert cfg.type in valid, f"invalid tokenizer type {cfg.type!r}, valid: {valid}"

Type guard

def is_valid_tokenizer_type(name: str) -> bool:
    from graphrag_llm.tokenizer.tokenizer_factory import tokenizer_factory
    return name in tokenizer_factory.keys()

Try / catch

try:
    tok = get_tokenizer(cfg)
except ValueError as e:
    raise ConfigError(str(e)) from e

Prevention

Use TokenizerType enum members instead of strings
Fail fast: validate the factory keys during config loading

When it happens

Trigger: Calling create_completion, create_embedding, or get_tokenizer with a TokenizerConfig whose type is misspelled, wrongly cased, or names an unsupported tokenizer (e.g. 'sentencepiece', 'huggingface').

Common situations: Hand-edited YAML/env tokenizer config, version drift after a strategy rename, or assuming a tokenizer backend exists because another library supports it.

Related errors

AI-assisted analysis of microsoft/graphrag@f40e9a26ce (2026-08-27). Data as JSON: /api/errors/002f2a13fb55c822. Report an issue: GitHub.