zylon-ai/private-gpt · error · NotImplementedError

TikTokenTokenizer only supports text token counting

Error message

TikTokenTokenizer only supports text token counting

What it means

Raised by TikTokenTokenizer.tokenize when the call includes images or audios. tiktoken encodings are text-only BPE tokenizers, so the implementation deletes the text-only options (add_special_tokens, truncation, max_length) and rejects any non-text payload with NotImplementedError. Multimodal inputs must be counted by a multimodal-aware component, not this class.

Source

Thrown at private_gpt/components/llm/tokenizers/tiktoken.py:107

    @property
    def is_multimodal(self) -> bool:
        return False

    def __call__(
        self,
        texts: TextLike | None = None,
        images: ImageLike | None = None,
        audios: AudioLike | None = None,
        add_special_tokens: bool = True,
        truncation: bool = False,
        max_length: int | None = None,
        **kwargs: Any,
    ) -> TokenizedInput:
        del add_special_tokens, truncation, max_length, kwargs

        if images or audios:
            raise NotImplementedError(
                "TikTokenTokenizer only supports text token counting"
            )
        if texts is None:
            return TokenizedInput(input_ids=[])

        if isinstance(texts, str):
            return TokenizedInput(input_ids=self._encoding.encode(texts))

        if isinstance(texts, Sequence):
            input_ids: list[int] = []
            for text in texts:
                input_ids.extend(self._encoding.encode(str(text)))
            return TokenizedInput(input_ids=input_ids)

        return TokenizedInput(input_ids=self._encoding.encode(str(texts)))

    def get_vocab(self) -> dict[str, int]:
        raise NotImplementedError(

View on GitHub (pinned to 4a030776a3)

Solutions

1. Pass only texts to tokenize(); count images/audios with a modality-specific counter (e.g. image tile estimator).
2. Make call sites forward images/audios only when the tokenizer advertises support.
3. Switch to a tokenizer implementation that supports your modalities if text-only counting is insufficient.

Example fix

# before
n = len(tok.tokenize(texts=msg, images=msg.images).input_ids)

# after
text_tokens = tok.tokenize(texts=msg).input_ids
image_tokens = estimate_image_tokens(msg.images)  # modality-specific

if images or audios:
    raise SkipModality('tiktoken tokenizer is text-only')
result = tok.tokenize(texts=texts)

def is_text_only_tokenizer(tok) -> bool:
    return type(tok).__name__ == 'TikTokenTokenizer'

try:
    res = tok.tokenize(texts=t, images=imgs)
except NotImplementedError:
    res = tok.tokenize(texts=t)

When it happens

Trigger: Calling tok.tokenize(texts=..., images=[...]) or tok.tokenize(texts=..., audios=[...]) on a TikTokenTokenizer; a token-counting helper that forwards every modality kwarg it receives; tests feeding multimodal fixtures to the tokenizer.

Common situations: Multimodal ingestion pipelines counting tokens for image+text pairs; shared abstraction calling one tokenizer for all content types; migrating from a multimodal tokenizer to tiktoken without trimming call sites.

Related errors

• RemoteTokenizeTokenizer only supports text tokenization
• Multimodal input provided but tokenizer is not multimodal
• tiktoken dependencies are not installed. Install with `uv sy
• LLM does not support structured chat.
• Audio blocks found but no audio-capable LLM provided.

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/225f9f9ca651ad55. Report an issue: GitHub.