huggingface/transformers · error · Exception

You're trying to run a `Unigram` model but you're file was t

Error message

You're trying to run a `Unigram` model but you're file was trained with a different algorithm

What it means

Generic Exception raised inside SPMConverter.tokenizer(): the sentencepiece trainer_spec.model_type is neither 1 (Unigram) nor 2 (BPE), so the converter has no strategy to build a fast tokenizer. The (historically worded) message says you are running a Unigram converter against a file trained with a different algorithm (in practice: char/word models, model_type 3+).

Source

Thrown at src/transformers/convert_slow_tokenizer.py:752

                )
            )

        elif model_type == 2:
            _, merges = self.SpmExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)
            bpe_vocab = {word: i for i, (word, score) in enumerate(vocab_scores)}
            tokenizer = Tokenizer(
                BPE(
                    bpe_vocab,
                    merges,
                    unk_token=proto.trainer_spec.unk_piece,
                    fuse_unk=True,
                    byte_fallback=self.handle_byte_fallback,
                    dropout=None,
                )
            )

        else:
            raise Exception(
                "You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
            )

        # control tokens are special
        # user defined symbols are not
        # both user and control tokens are AddedTokens
        # Add user defined symbols (type == 4) from sentencepiece (https://github.com/google/sentencepiece/blob/6225e08edb2577757163b3f5dbba4c0b670ef445/src/sentencepiece_model.proto#L299C29-L299C33)
        spm_added_tokens = [
            (id, p.piece, p.type == 3 or p.piece in self.special_tokens)
            for id, p in enumerate(proto.pieces)
            if p.type in [3, 4]
        ]
        tokenizer.add_tokens(
            [
                AddedToken(token, normalized=False, special=special)
                for id, token, special in sorted(spm_added_tokens, key=lambda x: x[0])
            ]
        )

View on GitHub (pinned to a597f97485)

Solutions

1. Inspect the model: python -c "from sentencepiece import SentencePieceProcessor; print(SentencePieceProcessor(model_file='t.model'))" or parse trainer_spec.model_type; if it is not UNIGRAM/BPE, no fast conversion exists.
2. Load with use_fast=False to keep using the slow tokenizer.
3. Retrain the tokenizer with sentencepiece in BPE or Unigram mode if a fast version is required.

Example fix

// before
tok = AutoTokenizer.from_pretrained("./char_sp_model_dir", use_fast=True)  # Exception

// after
tok = AutoTokenizer.from_pretrained("./char_sp_model_dir", use_fast=False)

import sentencepiece_model_pb2_new as pb2  # version-appropriate pb2
proto = pb2.ModelProto(); proto.ParseFromString(open(vocab_file, 'rb').read())
if proto.trainer_spec.model_type not in (1, 2):  # UNIGRAM, BPE
    use_fast = False  # no fast conversion path exists

def spm_model_is_convertible(vocab_file: str) -> bool:
    proto = load_proto(vocab_file)
    return proto.trainer_spec.model_type in (1, 2)

try:
    tok = AutoTokenizer.from_pretrained(path, use_fast=True)
except Exception as e:
    if "different algorithm" in str(e):
        tok = AutoTokenizer.from_pretrained(path, use_fast=False)

When it happens

Trigger: Calling convert_slow_tokenizer (directly or via AutoTokenizer.from_pretrained(..., use_fast=True)) on a sentencepiece .model file whose trainer was CHAR or WORD type, or a proto whose model_type enum is out of range (corrupt/unusual file).

Common situations: Very old or custom-trained sentencepiece models; proto files from other toolkits that happen to parse; conversions of tokenizers never intended for fast-tokenizer export.

Related errors

• {error_message} requires the protobuf library but it was not
• `tiktoken` is required to read a `tiktoken` file. Install it
• Converting from SentencePiece and Tiktoken failed, if a conv
• Unrecognized tokenizer name, should be one of {list(TOKENIZE
• Invalid checkpoint path: '{checkpoint}' attempts to escape `

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/7bb0d360493d83ba. Report an issue: GitHub.