{"record":{"id":"8dad6de173c4d4ff","repo":"headroomlabs-ai/headroom","slug":"self-class-name-does-not-support-encodin","errorCode":null,"errorMessage":"{self.__class__.__name__} does not support encoding","messagePattern":"(.+?) does not support encoding","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"warning","filePath":"headroom/tokenizers/base.py","lineNumber":441,"sourceCode":"        total += self.count_text(coerce_countable_text(function_call.get(\"arguments\")))\n        return total\n\n    def encode(self, text: str) -> list[int]:\n        \"\"\"Encode text to token IDs.\n\n        Optional method - not all backends support encoding.\n        Default implementation raises NotImplementedError.\n\n        Args:\n            text: Text to encode.\n\n        Returns:\n            List of token IDs.\n\n        Raises:\n            NotImplementedError: If encoding is not supported.\n        \"\"\"\n        raise NotImplementedError(f\"{self.__class__.__name__} does not support encoding\")\n\n    def decode(self, tokens: list[int]) -> str:\n        \"\"\"Decode token IDs to text.\n\n        Optional method - not all backends support decoding.\n        Default implementation raises NotImplementedError.\n\n        Args:\n            tokens: List of token IDs.\n\n        Returns:\n            Decoded text.\n\n        Raises:\n            NotImplementedError: If decoding is not supported.\n        \"\"\"\n        raise NotImplementedError(f\"{self.__class__.__name__} does not support decoding\")\n","sourceCodeStart":423,"sourceCodeEnd":459,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/tokenizers/base.py#L423-L459","documentation":"BaseTokenizer.encode() is an optional capability: the base class default raises NotImplementedError naming the subclass. Counting tokens (the core API) is always available, but round-tripping text to token IDs is only implemented by backends with real vocabularies (tiktoken, HuggingFace, etc.).","triggerScenarios":"Calling encode() on an estimating/heuristic tokenizer (e.g. EstimatingTokenCounter or a character-approximation backend) selected via the registry when no real backend is installed or the model is unknown.","commonSituations":"Using headroom in an environment without tiktoken/transformers installed; obscure or internal model names that fall through backend detection to the estimator; code that assumed encode() exists because it worked with another backend.","solutions":["Install a real backend (pip install tiktoken, or transformers for HuggingFace) so the registry resolves to an encoding-capable tokenizer.","Use a supported model name/tokenizer_name so backend detection picks tiktoken/HF instead of the estimator.","Only use count()/count_message_tokens() APIs if you do not need token IDs.","Check capability before calling: has_tokenizer() / is_available() classmethods where exposed."],"exampleFix":"# before\ntok = tokenizer_registry.get(\"internal-model-x\")\nids = tok.encode(\"hello\")  # NotImplementedError\n\n# after\nfrom headroom.tokenizers.tiktoken_counter import TiktokenTokenCounter\ntok = TiktokenTokenCounter(encoding_name=\"cl100k_base\")\nids = tok.encode(\"hello\")","handlingStrategy":"try-catch","validationCode":"encodable = hasattr(tokenizer, \"encode\") and type(tokenizer).encode is not BaseTokenizer.encode","typeGuard":"def supports_encoding(t) -> bool:\n    try:\n        t.encode(\"\")\n        return True\n    except NotImplementedError:\n        return False","tryCatchPattern":"try:\n    ids = tokenizer.encode(text)\nexcept NotImplementedError:\n    ids = None  # estimation-only backend; rely on count()","preventionTips":["Install real tokenizer backends where encode/decode are needed.","Feature-detect with a tiny probe before batch processing.","Design code paths to work off count() when IDs are unavailable."],"tags":["tokenizer","capability","not-implemented","dependencies"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}