{"record":{"id":"6758ef74860a0371","repo":"huggingface/tokenizers","slug":"you-don-t-seem-to-have-the-required-protobuf-file","errorCode":null,"errorMessage":"You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required.","messagePattern":"You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw\\.githubusercontent\\.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2\\.py` for us to be able to read the intrinsics of your spm_file\\. `pip install sentencepiece` is not required\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py","lineNumber":155,"sourceCode":"            unk_token=unk_token,\n        )\n\n        self._tokenizer.train_from_iterator(\n            iterator,\n            trainer=trainer,\n            length=length,\n        )\n\n    @staticmethod\n    def from_spm(filename: str):\n        try:\n            import sys\n\n            sys.path.append(\".\")\n\n            import sentencepiece_model_pb2 as model  # type: ignore[import]\n        except Exception:\n            raise Exception(\n                \"You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required.\"\n            )\n\n        m = model.ModelProto()\n        m.ParseFromString(open(filename, \"rb\").read())\n\n        precompiled_charsmap = m.normalizer_spec.precompiled_charsmap\n        vocab = [(piece.piece, piece.score) for piece in m.pieces]\n        unk_id = m.trainer_spec.unk_id\n        model_type = m.trainer_spec.model_type\n        byte_fallback = m.trainer_spec.byte_fallback\n        if model_type != 1:\n            raise Exception(\n                \"You're trying to run a `Unigram` model but you're file was trained with a different algorithm\"\n            )\n\n        replacement = \"▁\"\n        add_prefix_space = True","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py#L137-L173","documentation":"`Unigram.from_spm` needs to import `sentencepiece_model_pb2` (the SentencePiece protobuf schema) to parse the `.model` file. If that import fails — the generated `sentencepiece_model_pb2.py` is not on `sys.path` and the `sentencepiece` package isn't installed — this generic `Exception` is raised with instructions to install `protobuf` and download the pb2 file. The `sentencepiece` library itself is not required, only the protobuf definitions.","triggerScenarios":"Calling `Unigram.from_spm(\"tokenizer.model\")` in an environment where neither the `sentencepiece` pip package nor a manually downloaded `sentencepiece_model_pb2.py` in the working directory is available.","commonSituations":"Converting SentencePiece models (e.g. for T5/XLNet/ALBERT tokenizers) in clean CI or Docker images; the pb2 file was downloaded but into a different directory than the one on `sys.path` (the code appends `\".\"` only); protobuf version 4+/5+ incompatibilities breaking the old generated pb2 file.","solutions":["Run `pip install sentencepiece` (which ships the protobuf definitions) — or `pip install protobuf` plus downloading `sentencepiece_model_pb2.py` into the working directory per the error message.","Ensure `sentencepiece_model_pb2.py` is in the current working directory (the code does `sys.path.append(\".\")`), not just anywhere on PYTHONPATH.","If a downloaded pb2 fails due to protobuf version mismatch, regenerate it: `protoc --python_out=. sentencepiece_model.proto`."],"exampleFix":"// before (shell)\npython -c \"from tokenizers.implementations import Unigram; Unigram.from_spm('sp.model')\"\n// after (shell)\npip install sentencepiece\npython -c \"from tokenizers.implementations import Unigram; Unigram.from_spm('sp.model')\"","handlingStrategy":"fallback","validationCode":"import importlib.util\nif importlib.util.find_spec(\"sentencepiece\") is None and importlib.util.find_spec(\"sentencepiece_model_pb2\") is None:\n    raise ImportError(\"Install sentencepiece (pip install sentencepiece) or place sentencepiece_model_pb2.py in the working directory\")","typeGuard":"def can_load_spm() -> bool:\n    import importlib.util\n    return importlib.util.find_spec(\"sentencepiece\") is not None or importlib.util.find_spec(\"sentencepiece_model_pb2\") is not None","tryCatchPattern":"try:\n    uni = Unigram.from_spm(spm_file)\nexcept Exception as e:\n    if \"required protobuf file\" in str(e):\n        import subprocess\n        subprocess.run([\"pip\", \"install\", \"sentencepiece\"], check=True)\n        uni = Unigram.from_spm(spm_file)\n    else:\n        raise","preventionTips":["Add `sentencepiece` to requirements for any pipeline that converts SPM models.","If using the manual pb2 file, keep it next to the entry script (the code appends '.' to sys.path).","In CI/Docker, install protobuf and the pb2 file in the image build step, not at runtime."],"tags":["python","protobuf","missing-dependency","sentencepiece"],"backgroundTag":"missing-optional-dependency","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}