{"record":{"id":"89b4be62a277ec41","repo":"huggingface/tokenizers","slug":"you-re-trying-to-run-a-unigram-model-but-you-re","errorCode":null,"errorMessage":"You're trying to run a `Unigram` model but you're file was trained with a different algorithm","messagePattern":"You're trying to run a `Unigram` model but you're file was trained with a different algorithm","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py","lineNumber":168,"sourceCode":"\n            sys.path.append(\".\")\n\n            import sentencepiece_model_pb2 as model  # type: ignore[import]\n        except Exception:\n            raise Exception(\n                \"You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required.\"\n            )\n\n        m = model.ModelProto()\n        m.ParseFromString(open(filename, \"rb\").read())\n\n        precompiled_charsmap = m.normalizer_spec.precompiled_charsmap\n        vocab = [(piece.piece, piece.score) for piece in m.pieces]\n        unk_id = m.trainer_spec.unk_id\n        model_type = m.trainer_spec.model_type\n        byte_fallback = m.trainer_spec.byte_fallback\n        if model_type != 1:\n            raise Exception(\n                \"You're trying to run a `Unigram` model but you're file was trained with a different algorithm\"\n            )\n\n        replacement = \"▁\"\n        add_prefix_space = True\n\n        tokenizer = Tokenizer(Unigram(vocab, unk_id, byte_fallback))\n\n        if precompiled_charsmap:\n            tokenizer.normalizer = normalizers.Sequence(\n                [\n                    normalizers.Precompiled(precompiled_charsmap),\n                    normalizers.Replace(Regex(\" {2,}\"), \" \"),\n                ]\n            )\n        else:\n            tokenizer.normalizer = normalizers.Sequence([normalizers.Replace(Regex(\" {2,}\"), \" \")])\n        prepend_scheme = \"always\" if add_prefix_space else \"never\"","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py#L150-L186","documentation":"`Unigram.from_spm` parses the SentencePiece protobuf and checks `trainer_spec.model_type`. SentencePiece model type 1 is Unigram; if the file was trained with BPE (2), Word (3) or Char (4), the tokenizers library cannot load it as a `Unigram` model and raises this `Exception`. It's a model/algorithm mismatch, not a file corruption issue.","triggerScenarios":"Calling `Unigram.from_spm(\"model.model\")` on a SentencePiece model trained with `--model_type=bpe`, `word`, or `char` instead of `unigram`. Common with files trained for Llama/T5-style BPE exports.","commonSituations":"Assuming every `.model` SentencePiece file is Unigram (many community checkpoints use BPE); converting a SentencePiece model whose trainer spec was left at defaults; scripting bulk conversion of mixed-type SPM models.","solutions":["Retrain the SentencePiece model with `--model_type=unigram`, then use `Unigram.from_spm` on the new file.","Check the type before loading: `m.trainer_spec.model_type` via the pb2 module (1=UNIGRAM, 2=BPE, 3=WORD, 4=CHAR), and use the matching tokenizers model.","For BPE SPM files, use a converter that supports them (e.g. `transformers.convert_slow_tokenizer` / `Converter` for that architecture) instead of `Unigram.from_spm`."],"exampleFix":"// before (shell)\nspm_train --input=corpus.txt --model_prefix=sp --vocab_size=32000  # defaults to unigram? ensure explicit\n// after (shell)\nspm_train --input=corpus.txt --model_prefix=sp --model_type=unigram --vocab_size=32000","handlingStrategy":"validation","validationCode":"from sentencepiece import sentencepiece_model_pb2 as sp_pb2\nm = sp_pb2.ModelProto()\nm.ParseFromString(open(spm_file, \"rb\").read())\nif m.trainer_spec.model_type != sp_pb2.TrainerSpec.UNIGRAM:\n    raise ValueError(f\"SPM file is model_type={m.trainer_spec.model_type}, not UNIGRAM(1)\")\nuni = Unigram.from_spm(spm_file)","typeGuard":"def is_unigram_spm(spm_file: str) -> bool:\n    import sentencepiece as spm\n    s = spm.SentencePieceProcessor()\n    s.Load(spm_file)\n    return True  # inspect type via pb2 before calling Unigram.from_spm","tryCatchPattern":"try:\n    uni = Unigram.from_spm(spm_file)\nexcept Exception as e:\n    if \"different algorithm\" in str(e):\n        raise ValueError(f\"{spm_file} is not a Unigram SPM model; retrain with --model_type=unigram\") from e\n    raise","preventionTips":["Record model_type when training SentencePiece models and assert it before conversion.","Audit bulk conversion jobs: check trainer_spec.model_type for each .model file.","Default spm_train output may differ from expectations — always pass --model_type explicitly."],"tags":["python","sentencepiece","model-mismatch","unigram"],"backgroundTag":"incompatible-source-type","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}