microsoft/graphrag · error · ValueError

TokenizerConfig.type

Error message

TokenizerConfig.type '{strategy}' is not registered in the TokenizerFactory. Registered strategies: {', '.join(tokenizer_factory.keys())}

What it means

The tokenizer factory registers a fixed set of strategies (e.g. Tiktoken) in its match statement. Setting TokenizerConfig.type to any unregistered name hits the wildcard case and raises ValueError listing valid strategies.

Solutions

Set TokenizerConfig.type to one of the strategies printed in the error message
Verify exact casing/whitespace in the config source
Register the custom tokenizer in TokenizerFactory if you genuinely need a new backend
Align the graphrag-llm version with the config schema you followed

Example fix

# before
cfg = TokenizerConfig(type='hf')
tok = get_tokenizer(cfg)

# after
from graphrag_llm.tokenizer import TokenizerType
cfg = TokenizerConfig(type=TokenizerType.Tiktoken)
tok = get_tokenizer(cfg)

Defensive patterns

Strategy: validation

Validate before calling

from graphrag_llm.tokenizer.tokenizer_factory import tokenizer_factory
valid = set(tokenizer_factory.keys())
assert cfg.type in valid, f"invalid tokenizer type {cfg.type!r}, valid: {valid}"

Type guard

def is_valid_tokenizer_type(name: str) -> bool:
    from graphrag_llm.tokenizer.tokenizer_factory import tokenizer_factory
    return name in tokenizer_factory.keys()

Try / catch

try:
    tok = get_tokenizer(cfg)
except ValueError as e:
    raise ConfigError(str(e)) from e

Prevention

Use TokenizerType enum members instead of strings
Fail fast: validate the factory keys during config loading

When it happens

Trigger: Calling create_completion, create_embedding, or get_tokenizer with a TokenizerConfig whose type is misspelled, wrongly cased, or names an unsupported tokenizer (e.g. 'sentencepiece', 'huggingface').

Common situations: Hand-edited YAML/env tokenizer config, version drift after a strategy rename, or assuming a tokenizer backend exists because another library supports it.

Related errors

AI-assisted analysis of microsoft/graphrag@f40e9a26ce (2026-08-27). Data as JSON: /api/errors/002f2a13fb55c822. Report an issue: GitHub.

Appendix: source

Thrown at packages/graphrag-llm/graphrag_llm/tokenizer/tokenizer_factory.py:84

                register_tokenizer(
                    TokenizerType.LiteLLM,
                    LiteLLMTokenizer,
                    scope="singleton",
                )
            case TokenizerType.Tiktoken:
                from graphrag_llm.tokenizer.tiktoken_tokenizer import (
                    TiktokenTokenizer,
                )

                register_tokenizer(
                    TokenizerType.Tiktoken,
                    TiktokenTokenizer,
                    scope="singleton",
                )
            case _:
                msg = f"TokenizerConfig.type '{strategy}' is not registered in the TokenizerFactory. Registered strategies: {', '.join(tokenizer_factory.keys())}"
                raise ValueError(msg)

    return tokenizer_factory.create(
        strategy=strategy,
        init_args=init_args,
    )

View on GitHub (pinned to f40e9a26ce)