huggingface/tokenizers · error · Exception

You're trying to run a `Unigram` model but you're file was…

Error message

You're trying to run a `Unigram` model but you're file was trained with a different algorithm

What it means

`Unigram.from_spm` parses the SentencePiece protobuf and checks `trainer_spec.model_type`. SentencePiece model type 1 is Unigram; if the file was trained with BPE (2), Word (3) or Char (4), the tokenizers library cannot load it as a `Unigram` model and raises this `Exception`. It's a model/algorithm mismatch, not a file corruption issue.

Solutions

1. Retrain the SentencePiece model with `--model_type=unigram`, then use `Unigram.from_spm` on the new file.
2. Check the type before loading: `m.trainer_spec.model_type` via the pb2 module (1=UNIGRAM, 2=BPE, 3=WORD, 4=CHAR), and use the matching tokenizers model.
3. For BPE SPM files, use a converter that supports them (e.g. `transformers.convert_slow_tokenizer` / `Converter` for that architecture) instead of `Unigram.from_spm`.

Example fix

// before (shell)
spm_train --input=corpus.txt --model_prefix=sp --vocab_size=32000  # defaults to unigram? ensure explicit
// after (shell)
spm_train --input=corpus.txt --model_prefix=sp --model_type=unigram --vocab_size=32000

from sentencepiece import sentencepiece_model_pb2 as sp_pb2
m = sp_pb2.ModelProto()
m.ParseFromString(open(spm_file, "rb").read())
if m.trainer_spec.model_type != sp_pb2.TrainerSpec.UNIGRAM:
    raise ValueError(f"SPM file is model_type={m.trainer_spec.model_type}, not UNIGRAM(1)")
uni = Unigram.from_spm(spm_file)

def is_unigram_spm(spm_file: str) -> bool:
    import sentencepiece as spm
    s = spm.SentencePieceProcessor()
    s.Load(spm_file)
    return True  # inspect type via pb2 before calling Unigram.from_spm

try:
    uni = Unigram.from_spm(spm_file)
except Exception as e:
    if "different algorithm" in str(e):
        raise ValueError(f"{spm_file} is not a Unigram SPM model; retrain with --model_type=unigram") from e
    raise

When it happens

Trigger: Calling `Unigram.from_spm("model.model")` on a SentencePiece model trained with `--model_type=bpe`, `word`, or `char` instead of `unigram`. Common with files trained for Llama/T5-style BPE exports.

Common situations: Assuming every `.model` SentencePiece file is Unigram (many community checkpoints use BPE); converting a SentencePiece model whose trainer spec was left at defaults; scripting bulk conversion of mixed-type SPM models.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• You don't seem to have the required protobuf file, in order…
• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/89b4be62a277ec41. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py:168

            sys.path.append(".")

            import sentencepiece_model_pb2 as model  # type: ignore[import]
        except Exception:
            raise Exception(
                "You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required."
            )

        m = model.ModelProto()
        m.ParseFromString(open(filename, "rb").read())

        precompiled_charsmap = m.normalizer_spec.precompiled_charsmap
        vocab = [(piece.piece, piece.score) for piece in m.pieces]
        unk_id = m.trainer_spec.unk_id
        model_type = m.trainer_spec.model_type
        byte_fallback = m.trainer_spec.byte_fallback
        if model_type != 1:
            raise Exception(
                "You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
            )

        replacement = "▁"
        add_prefix_space = True

        tokenizer = Tokenizer(Unigram(vocab, unk_id, byte_fallback))

        if precompiled_charsmap:
            tokenizer.normalizer = normalizers.Sequence(
                [
                    normalizers.Precompiled(precompiled_charsmap),
                    normalizers.Replace(Regex(" {2,}"), " "),
                ]
            )
        else:
            tokenizer.normalizer = normalizers.Sequence([normalizers.Replace(Regex(" {2,}"), " ")])
        prepend_scheme = "always" if add_prefix_space else "never"

View on GitHub (pinned to 6cfd9d385c)