huggingface/tokenizers · error · Exception

You don't seem to have the required protobuf file, in order…

Error message

You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required.

What it means

`Unigram.from_spm` needs to import `sentencepiece_model_pb2` (the SentencePiece protobuf schema) to parse the `.model` file. If that import fails — the generated `sentencepiece_model_pb2.py` is not on `sys.path` and the `sentencepiece` package isn't installed — this generic `Exception` is raised with instructions to install `protobuf` and download the pb2 file. The `sentencepiece` library itself is not required, only the protobuf definitions.

Solutions

1. Run `pip install sentencepiece` (which ships the protobuf definitions) — or `pip install protobuf` plus downloading `sentencepiece_model_pb2.py` into the working directory per the error message.
2. Ensure `sentencepiece_model_pb2.py` is in the current working directory (the code does `sys.path.append(".")`), not just anywhere on PYTHONPATH.
3. If a downloaded pb2 fails due to protobuf version mismatch, regenerate it: `protoc --python_out=. sentencepiece_model.proto`.

Example fix

// before (shell)
python -c "from tokenizers.implementations import Unigram; Unigram.from_spm('sp.model')"
// after (shell)
pip install sentencepiece
python -c "from tokenizers.implementations import Unigram; Unigram.from_spm('sp.model')"

import importlib.util
if importlib.util.find_spec("sentencepiece") is None and importlib.util.find_spec("sentencepiece_model_pb2") is None:
    raise ImportError("Install sentencepiece (pip install sentencepiece) or place sentencepiece_model_pb2.py in the working directory")

def can_load_spm() -> bool:
    import importlib.util
    return importlib.util.find_spec("sentencepiece") is not None or importlib.util.find_spec("sentencepiece_model_pb2") is not None

try:
    uni = Unigram.from_spm(spm_file)
except Exception as e:
    if "required protobuf file" in str(e):
        import subprocess
        subprocess.run(["pip", "install", "sentencepiece"], check=True)
        uni = Unigram.from_spm(spm_file)
    else:
        raise

When it happens

Trigger: Calling `Unigram.from_spm("tokenizer.model")` in an environment where neither the `sentencepiece` pip package nor a manually downloaded `sentencepiece_model_pb2.py` in the working directory is available.

Common situations: Converting SentencePiece models (e.g. for T5/XLNet/ALBERT tokenizers) in clean CI or Docker images; the pb2 file was downloaded but into a different directory than the one on `sys.path` (the code appends `"."` only); protobuf version 4+/5+ incompatibilities breaking the old generated pb2 file.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• You're trying to run a `Unigram` model but you're file was…
• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/6758ef74860a0371. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py:155

            unk_token=unk_token,
        )

        self._tokenizer.train_from_iterator(
            iterator,
            trainer=trainer,
            length=length,
        )

    @staticmethod
    def from_spm(filename: str):
        try:
            import sys

            sys.path.append(".")

            import sentencepiece_model_pb2 as model  # type: ignore[import]
        except Exception:
            raise Exception(
                "You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required."
            )

        m = model.ModelProto()
        m.ParseFromString(open(filename, "rb").read())

        precompiled_charsmap = m.normalizer_spec.precompiled_charsmap
        vocab = [(piece.piece, piece.score) for piece in m.pieces]
        unk_id = m.trainer_spec.unk_id
        model_type = m.trainer_spec.model_type
        byte_fallback = m.trainer_spec.byte_fallback
        if model_type != 1:
            raise Exception(
                "You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
            )

        replacement = "▁"
        add_prefix_space = True

View on GitHub (pinned to 6cfd9d385c)