{"record":{"id":"529ab869136885a1","repo":"headroomlabs-ai/headroom","slug":"self-class-name-does-not-support-decodin","errorCode":null,"errorMessage":"{self.__class__.__name__} does not support decoding","messagePattern":"(.+?) does not support decoding","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"warning","filePath":"headroom/tokenizers/base.py","lineNumber":458,"sourceCode":"        \"\"\"\n        raise NotImplementedError(f\"{self.__class__.__name__} does not support encoding\")\n\n    def decode(self, tokens: list[int]) -> str:\n        \"\"\"Decode token IDs to text.\n\n        Optional method - not all backends support decoding.\n        Default implementation raises NotImplementedError.\n\n        Args:\n            tokens: List of token IDs.\n\n        Returns:\n            Decoded text.\n\n        Raises:\n            NotImplementedError: If decoding is not supported.\n        \"\"\"\n        raise NotImplementedError(f\"{self.__class__.__name__} does not support decoding\")\n\n\nclass _DelegatingBlockCounter(BaseTokenizer):\n    \"\"\"Adapter exposing :meth:`BaseTokenizer._count_content_parts` to non-subclasses.\n\n    The provider token counters in ``headroom/providers/`` are not\n    ``BaseTokenizer`` subclasses, and each grew its own shortened content-block\n    walker that handled only the shapes its provider was expected to send. The\n    result was that every one of them priced most modern blocks at ~0: measured\n    on a 6,800-char block, ``OpenAITokenCounter`` returned 8 tokens for\n    ``tool_result``/``thinking``/``document``/``mcp_tool_result`` and — its own\n    Responses shapes — ``output_text``/``refusal``; ``AnthropicTokenCounter``\n    returned 7 for ``thinking``/``document``, which are Anthropic's own.\n\n    Rather than add a fifth partial walker, this lets them borrow the audited one.\n    It is image-safe (base64 blobs get a pixel-based estimate instead of being\n    serialized and priced as text) and bounds oversized blobs, which a naive\n    ``count_text(str(block))`` catch-all does not.","sourceCodeStart":440,"sourceCodeEnd":476,"githubUrl":"https://github.com/headroomlabs-ai/headroom/blob/322425c43bffde1ed0b64fecf3cf5951565dd82b/headroom/tokenizers/base.py#L440-L476","documentation":"BaseTokenizer.decode() is optional: the base-class default raises NotImplementedError naming the subclass, because not every backend can map token IDs back to text. Estimating/heuristic counters only approximate counts and carry no vocabulary, so decode is impossible.","triggerScenarios":"Calling decode() on an EstimatingTokenCounter or any fallback tokenizer obtained from the registry when the real backend was unavailable (missing dependency, unknown model, failed load).","commonSituations":"Environments without tiktoken/transformers installed; fallback kicked in after a tokenizer load failure (see HuggingFace fallback errors) and code still tries to decode; tests written against tiktoken running on minimal CI images.","solutions":["Install the backend dependency (tiktoken / transformers) and request a known model so the registry returns a real tokenizer.","Branch on capability: guard decode calls with isinstance checks against a decoding-capable class or a try/except NotImplementedError.","If decode is essential, pin a specific tokenizer explicitly (e.g. TiktokenTokenCounter) instead of relying on auto-detection."],"exampleFix":"# before\ntext = tokenizer.decode([9450, 1917])  # NotImplementedError on estimator\n\n# after\ntry:\n    text = tokenizer.decode([9450, 1917])\nexcept NotImplementedError:\n    text = None  # estimation-only backend; skip round-trip logic","handlingStrategy":"try-catch","validationCode":"decodable = hasattr(tokenizer, \"decode\") and type(tokenizer).decode is not BaseTokenizer.decode","typeGuard":"def supports_decoding(t) -> bool:\n    try:\n        t.decode([0])\n        return True\n    except NotImplementedError:\n        return False","tryCatchPattern":"try:\n    text = tokenizer.decode(ids)\nexcept NotImplementedError:\n    text = None","preventionTips":["Pin an encoding-capable tokenizer class when round-trips matter.","Catch NotImplementedError at the boundary and degrade gracefully.","Test decode paths against the exact backend used in production."],"tags":["tokenizer","capability","not-implemented","dependencies"],"backgroundTag":null,"analyzedSha":"322425c43bffde1ed0b64fecf3cf5951565dd82b","analyzedAt":"2026-08-15T01:03:05.481Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}