Comfy-Org/ComfyUI · error · ValueError

invalid tokenizer

Error message

invalid tokenizer

What it means

SPieceTokenizer accepts either raw SentencePiece model bytes (from a tensor) or a filesystem path to a .model file. When given a string that is not an existing file (bad path, folder, or typo), it raises 'invalid tokenizer' before constructing the SentencePieceProcessor. It is a load-time path validation, not a format check.

Source

Thrown at comfy/text_encoders/spiece_tokenizer.py:21

class SPieceTokenizer:
    @staticmethod
    def from_pretrained(path, **kwargs):
        return SPieceTokenizer(path, **kwargs)

    def __init__(self, tokenizer_path, add_bos=False, add_eos=True, special_tokens=None):
        self.add_bos = add_bos
        self.add_eos = add_eos
        self.special_tokens = special_tokens
        import sentencepiece
        if torch.is_tensor(tokenizer_path):
            tokenizer_path = tokenizer_path.numpy().tobytes()

        if isinstance(tokenizer_path, bytes):
            self.tokenizer = sentencepiece.SentencePieceProcessor(model_proto=tokenizer_path, add_bos=self.add_bos, add_eos=self.add_eos)
        else:
            if not os.path.isfile(tokenizer_path):
                raise ValueError("invalid tokenizer")
            self.tokenizer = sentencepiece.SentencePieceProcessor(model_file=tokenizer_path, add_bos=self.add_bos, add_eos=self.add_eos)

    def get_vocab(self):
        out = {}
        for i in range(self.tokenizer.get_piece_size()):
            out[self.tokenizer.id_to_piece(i)] = i
        return out

    def __call__(self, string):
        if self.special_tokens is not None:
            import re
            special_tokens_pattern = '|'.join(re.escape(token) for token in self.special_tokens.keys())
            if special_tokens_pattern and re.search(special_tokens_pattern, string):
                parts = re.split(f'({special_tokens_pattern})', string)
                result = []
                for part in parts:
                    if not part:
                        continue

View on GitHub (pinned to 1c6d8d45b3)

Solutions

1. Verify the SentencePiece .model file exists at the path you pass (os.path.isfile check) and fix the path
2. Ensure the tokenizer file was downloaded/placed alongside the text encoder weights in models/text_encoders
3. If you have the tokenizer as bytes/tensor, pass bytes instead of a path so the isfile branch is skipped

Example fix

// before
tok = SPieceTokenizer('/models/llava/spiece.model')  # file missing

# after
path = os.path.join(embedding_dir, 'spiece.model')
if not os.path.isfile(path):
    raise FileNotFoundError(path)
tok = SPieceTokenizer(path)

import os
if isinstance(path, str) and not os.path.isfile(path):
    raise FileNotFoundError(f'tokenizer model not found: {path}')
tok = SPieceTokenizer(path)

def is_spiece_source(p) -> bool:
    return isinstance(p, (bytes, bytearray)) or (isinstance(p, str) and os.path.isfile(p))

When it happens

Trigger: Passing a str tokenizer_path that fails os.path.isfile — wrong path, missing tokenizer.model in models/text_encoders, path with typos, or the string path of a directory.

Common situations: Model folder renamed or tokenizer.model not downloaded; embedding_directory misconfigured in a custom node; passing a tokenizer path from a different machine with different layout.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• Lens tokenizer requires the ``tokenizer_json`` byte tensor i
• MiniMax Music3 text encoder checkpoint is missing tokenizer_
• MiniMax Music3 tokenizer mismatch for {token}
• Unknown patch method: {self.patch_method}
• Image-token count {image_idx.shape[0]} != ViT output count {

AI-assisted analysis of Comfy-Org/ComfyUI@1c6d8d45b3 (2026-08-14). Data as JSON: /api/errors/b6d6aa5c10770eca. Report an issue: GitHub.