{"record":{"id":"c8589c02739873f8","repo":"headroomlabs-ai/headroom","slug":"must-provide-either-tokenizer-or-factory","errorCode":null,"errorMessage":"Must provide either tokenizer or factory","messagePattern":"Must provide either tokenizer or factory","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"headroom/tokenizers/registry.py","lineNumber":240,"sourceCode":"        \"\"\"Register a tokenizer or factory for a model.\n\n        Args:\n            model: Model name to register.\n            tokenizer: Pre-instantiated tokenizer instance.\n            factory: Factory function that creates tokenizer for model.\n\n        Raises:\n            ValueError: If neither tokenizer nor factory provided.\n        \"\"\"\n        registry = cls()\n        model_lower = model.lower()\n\n        if tokenizer is not None:\n            registry._tokenizers[model_lower] = tokenizer\n        elif factory is not None:\n            registry._factories[model_lower] = factory\n        else:\n            raise ValueError(\"Must provide either tokenizer or factory\")\n\n        # Clear cache for this model\n        keys_to_remove = [k for k in registry._cache if k.startswith(model_lower)]\n        for key in keys_to_remove:\n            del registry._cache[key]\n\n    @classmethod\n    def register_backend(\n        cls,\n        backend: str,\n        factory: Callable[[str], TokenCounter],\n    ) -> None:\n        \"\"\"Register a backend factory.\n\n        Args:\n            backend: Backend name.\n            factory: Factory function (model: str) -> TokenCounter.\n        \"\"\"","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/tokenizers/registry.py#L222-L258","documentation":"TokenizerRegistry.register() requires exactly one of `tokenizer` (a ready TokenCounter instance) or `factory` (a callable model->TokenCounter); passing neither — e.g. register(model) with both None — raises this ValueError. The guard prevents registering a model name that could never produce a tokenizer.","triggerScenarios":"TokenizerRegistry.register('my-model') with no arguments; passing both None explicitly; a wrapper function that forwards optional args and drops them due to a typo'd kwarg (e.g. factor=, tokeniser=).","commonSituations":"Programmatic registration loops where some entries only carry metadata; refactors renaming parameters; copy-pasted registration code from examples with placeholders not filled in.","solutions":["Pass one of the two: register('my-model', tokenizer=MyCounter()) or register('my-model', factory=lambda m: MyCounter(m)).","Check for typos in keyword names so values are not silently dropped to None.","Validate registration inputs in test setup: assert tokenizer or factory is not None."],"exampleFix":"# before\nTokenizerRegistry.register(\"my-model\")  # ValueError\n\n# after\nTokenizerRegistry.register(\"my-model\", factory=lambda model: MyModelCounter(model))","handlingStrategy":"validation","validationCode":"assert tokenizer is not None or factory is not None, \"register needs one of tokenizer/factory\"\nTokenizerRegistry.register(model, tokenizer=tokenizer, factory=factory)","typeGuard":"def valid_registration(tokenizer, factory) -> bool:\n    return (tokenizer is not None) != (factory is not None)","tryCatchPattern":null,"preventionTips":["Pass exactly one of tokenizer/factory as a keyword.","Add a startup test that registers all custom models successfully.","Type-check registration wrappers with mypy."],"tags":["registry","validation","api-misuse"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}