headroomlabs-ai/headroom · error · ValueError

Must provide either tokenizer or factory

Error message

Must provide either tokenizer or factory

What it means

TokenizerRegistry.register() requires exactly one of `tokenizer` (a ready TokenCounter instance) or `factory` (a callable model->TokenCounter); passing neither — e.g. register(model) with both None — raises this ValueError. The guard prevents registering a model name that could never produce a tokenizer.

Source

Thrown at headroom/tokenizers/registry.py:240

        """Register a tokenizer or factory for a model.

        Args:
            model: Model name to register.
            tokenizer: Pre-instantiated tokenizer instance.
            factory: Factory function that creates tokenizer for model.

        Raises:
            ValueError: If neither tokenizer nor factory provided.
        """
        registry = cls()
        model_lower = model.lower()

        if tokenizer is not None:
            registry._tokenizers[model_lower] = tokenizer
        elif factory is not None:
            registry._factories[model_lower] = factory
        else:
            raise ValueError("Must provide either tokenizer or factory")

        # Clear cache for this model
        keys_to_remove = [k for k in registry._cache if k.startswith(model_lower)]
        for key in keys_to_remove:
            del registry._cache[key]

    @classmethod
    def register_backend(
        cls,
        backend: str,
        factory: Callable[[str], TokenCounter],
    ) -> None:
        """Register a backend factory.

        Args:
            backend: Backend name.
            factory: Factory function (model: str) -> TokenCounter.
        """

View on GitHub (pinned to 322425c43b)

Solutions

1. Pass one of the two: register('my-model', tokenizer=MyCounter()) or register('my-model', factory=lambda m: MyCounter(m)).
2. Check for typos in keyword names so values are not silently dropped to None.
3. Validate registration inputs in test setup: assert tokenizer or factory is not None.

Example fix

# before
TokenizerRegistry.register("my-model")  # ValueError

# after
TokenizerRegistry.register("my-model", factory=lambda model: MyModelCounter(model))

assert tokenizer is not None or factory is not None, "register needs one of tokenizer/factory"
TokenizerRegistry.register(model, tokenizer=tokenizer, factory=factory)

def valid_registration(tokenizer, factory) -> bool:
    return (tokenizer is not None) != (factory is not None)

When it happens

Trigger: TokenizerRegistry.register('my-model') with no arguments; passing both None explicitly; a wrapper function that forwards optional args and drops them due to a typo'd kwarg (e.g. factor=, tokeniser=).

Common situations: Programmatic registration loops where some entries only carry metadata; refactors renaming parameters; copy-pasted registration code from examples with placeholders not filled in.

Related errors

• Unknown backend: {backend}
• compressor descriptor.name must be non-empty
• invalid pipeline config TOML: {0}
• recommendations file not found: {0}
• bedrock_eventstream_parse_failed

AI-assisted analysis of headroomlabs-ai/headroom@322425c43b (2026-08-15). Data as JSON: /api/errors/c8589c02739873f8. Report an issue: GitHub.