{"record":{"id":"7bb0d360493d83ba","repo":"huggingface/transformers","slug":"you-re-trying-to-run-a-unigram-model-but-you-re","errorCode":null,"errorMessage":"You're trying to run a `Unigram` model but you're file was trained with a different algorithm","messagePattern":"You're trying to run a `Unigram` model but you're file was trained with a different algorithm","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"src/transformers/convert_slow_tokenizer.py","lineNumber":752,"sourceCode":"                )\n            )\n\n        elif model_type == 2:\n            _, merges = self.SpmExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)\n            bpe_vocab = {word: i for i, (word, score) in enumerate(vocab_scores)}\n            tokenizer = Tokenizer(\n                BPE(\n                    bpe_vocab,\n                    merges,\n                    unk_token=proto.trainer_spec.unk_piece,\n                    fuse_unk=True,\n                    byte_fallback=self.handle_byte_fallback,\n                    dropout=None,\n                )\n            )\n\n        else:\n            raise Exception(\n                \"You're trying to run a `Unigram` model but you're file was trained with a different algorithm\"\n            )\n\n        # control tokens are special\n        # user defined symbols are not\n        # both user and control tokens are AddedTokens\n        # Add user defined symbols (type == 4) from sentencepiece (https://github.com/google/sentencepiece/blob/6225e08edb2577757163b3f5dbba4c0b670ef445/src/sentencepiece_model.proto#L299C29-L299C33)\n        spm_added_tokens = [\n            (id, p.piece, p.type == 3 or p.piece in self.special_tokens)\n            for id, p in enumerate(proto.pieces)\n            if p.type in [3, 4]\n        ]\n        tokenizer.add_tokens(\n            [\n                AddedToken(token, normalized=False, special=special)\n                for id, token, special in sorted(spm_added_tokens, key=lambda x: x[0])\n            ]\n        )","sourceCodeStart":734,"sourceCodeEnd":770,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/convert_slow_tokenizer.py#L734-L770","documentation":"Generic Exception raised inside SPMConverter.tokenizer(): the sentencepiece trainer_spec.model_type is neither 1 (Unigram) nor 2 (BPE), so the converter has no strategy to build a fast tokenizer. The (historically worded) message says you are running a Unigram converter against a file trained with a different algorithm (in practice: char/word models, model_type 3+).","triggerScenarios":"Calling convert_slow_tokenizer (directly or via AutoTokenizer.from_pretrained(..., use_fast=True)) on a sentencepiece .model file whose trainer was CHAR or WORD type, or a proto whose model_type enum is out of range (corrupt/unusual file).","commonSituations":"Very old or custom-trained sentencepiece models; proto files from other toolkits that happen to parse; conversions of tokenizers never intended for fast-tokenizer export.","solutions":["Inspect the model: python -c \"from sentencepiece import SentencePieceProcessor; print(SentencePieceProcessor(model_file='t.model'))\" or parse trainer_spec.model_type; if it is not UNIGRAM/BPE, no fast conversion exists.","Load with use_fast=False to keep using the slow tokenizer.","Retrain the tokenizer with sentencepiece in BPE or Unigram mode if a fast version is required."],"exampleFix":"// before\ntok = AutoTokenizer.from_pretrained(\"./char_sp_model_dir\", use_fast=True)  # Exception\n\n// after\ntok = AutoTokenizer.from_pretrained(\"./char_sp_model_dir\", use_fast=False)","handlingStrategy":"type-guard","validationCode":"import sentencepiece_model_pb2_new as pb2  # version-appropriate pb2\nproto = pb2.ModelProto(); proto.ParseFromString(open(vocab_file, 'rb').read())\nif proto.trainer_spec.model_type not in (1, 2):  # UNIGRAM, BPE\n    use_fast = False  # no fast conversion path exists","typeGuard":"def spm_model_is_convertible(vocab_file: str) -> bool:\n    proto = load_proto(vocab_file)\n    return proto.trainer_spec.model_type in (1, 2)","tryCatchPattern":"try:\n    tok = AutoTokenizer.from_pretrained(path, use_fast=True)\nexcept Exception as e:\n    if \"different algorithm\" in str(e):\n        tok = AutoTokenizer.from_pretrained(path, use_fast=False)","preventionTips":["Check trainer_spec.model_type before promising fast-tokenizer support for custom sentencepiece models.","Keep a slow-tokenizer fallback in pipelines that accept arbitrary user checkpoints."],"tags":["tokenizer","sentencepiece","conversion"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}