{"record":{"id":"225f9f9ca651ad55","repo":"zylon-ai/private-gpt","slug":"tiktokentokenizer-only-supports-text-token-countin","errorCode":null,"errorMessage":"TikTokenTokenizer only supports text token counting","messagePattern":"TikTokenTokenizer only supports text token counting","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/llm/tokenizers/tiktoken.py","lineNumber":107,"sourceCode":"\n    @property\n    def is_multimodal(self) -> bool:\n        return False\n\n    def __call__(\n        self,\n        texts: TextLike | None = None,\n        images: ImageLike | None = None,\n        audios: AudioLike | None = None,\n        add_special_tokens: bool = True,\n        truncation: bool = False,\n        max_length: int | None = None,\n        **kwargs: Any,\n    ) -> TokenizedInput:\n        del add_special_tokens, truncation, max_length, kwargs\n\n        if images or audios:\n            raise NotImplementedError(\n                \"TikTokenTokenizer only supports text token counting\"\n            )\n        if texts is None:\n            return TokenizedInput(input_ids=[])\n\n        if isinstance(texts, str):\n            return TokenizedInput(input_ids=self._encoding.encode(texts))\n\n        if isinstance(texts, Sequence):\n            input_ids: list[int] = []\n            for text in texts:\n                input_ids.extend(self._encoding.encode(str(text)))\n            return TokenizedInput(input_ids=input_ids)\n\n        return TokenizedInput(input_ids=self._encoding.encode(str(texts)))\n\n    def get_vocab(self) -> dict[str, int]:\n        raise NotImplementedError(","sourceCodeStart":89,"sourceCodeEnd":125,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/llm/tokenizers/tiktoken.py#L89-L125","documentation":"Raised by TikTokenTokenizer.tokenize when the call includes images or audios. tiktoken encodings are text-only BPE tokenizers, so the implementation deletes the text-only options (add_special_tokens, truncation, max_length) and rejects any non-text payload with NotImplementedError. Multimodal inputs must be counted by a multimodal-aware component, not this class.","triggerScenarios":"Calling tok.tokenize(texts=..., images=[...]) or tok.tokenize(texts=..., audios=[...]) on a TikTokenTokenizer; a token-counting helper that forwards every modality kwarg it receives; tests feeding multimodal fixtures to the tokenizer.","commonSituations":"Multimodal ingestion pipelines counting tokens for image+text pairs; shared abstraction calling one tokenizer for all content types; migrating from a multimodal tokenizer to tiktoken without trimming call sites.","solutions":["Pass only texts to tokenize(); count images/audios with a modality-specific counter (e.g. image tile estimator).","Make call sites forward images/audios only when the tokenizer advertises support.","Switch to a tokenizer implementation that supports your modalities if text-only counting is insufficient."],"exampleFix":"# before\nn = len(tok.tokenize(texts=msg, images=msg.images).input_ids)\n\n# after\ntext_tokens = tok.tokenize(texts=msg).input_ids\nimage_tokens = estimate_image_tokens(msg.images)  # modality-specific","handlingStrategy":"validation","validationCode":"if images or audios:\n    raise SkipModality('tiktoken tokenizer is text-only')\nresult = tok.tokenize(texts=texts)","typeGuard":"def is_text_only_tokenizer(tok) -> bool:\n    return type(tok).__name__ == 'TikTokenTokenizer'","tryCatchPattern":"try:\n    res = tok.tokenize(texts=t, images=imgs)\nexcept NotImplementedError:\n    res = tok.tokenize(texts=t)","preventionTips":["Compute image/audio token budgets with dedicated estimators; reserve the text tokenizer for strings.","Assert call sites forward only supported kwargs for the configured mode."],"tags":["tokenizer","multimodal","not-implemented","tiktoken"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}