{"record":{"id":"af014127abd310dd","repo":"huggingface/transformers","slug":"error-message-requires-the-protobuf-library-but","errorCode":null,"errorMessage":"{error_message} requires the protobuf library but it was not found in your environment. Check out the instructions on the installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones that match your environment. Please note that you may need to restart your runtime after installation.","messagePattern":"(.+?) requires the protobuf library but it was not found in your environment\\. Check out the instructions on the installation page of its repo: https://github\\.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones that match your environment\\. Please note that you may need to restart your runtime after installation\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"src/transformers/convert_slow_tokenizer.py","lineNumber":109,"sourceCode":"    \"sl_SI\",\n]\n\n\ndef import_protobuf(error_message=\"\"):\n    if is_sentencepiece_available():\n        from sentencepiece import sentencepiece_model_pb2\n\n        return sentencepiece_model_pb2\n    if is_protobuf_available():\n        import google.protobuf\n\n        if version.parse(google.protobuf.__version__) < version.parse(\"4.0.0\"):\n            from transformers.utils import sentencepiece_model_pb2\n        else:\n            from transformers.utils import sentencepiece_model_pb2_new as sentencepiece_model_pb2\n        return sentencepiece_model_pb2\n    else:\n        raise ImportError(PROTOBUF_IMPORT_ERROR.format(error_message))\n\n\ndef _get_prepend_scheme(add_prefix_space: bool, original_tokenizer) -> str:\n    if add_prefix_space:\n        prepend_scheme = \"always\"\n        if not getattr(original_tokenizer, \"legacy\", True):\n            prepend_scheme = \"first\"\n    else:\n        prepend_scheme = \"never\"\n    return prepend_scheme\n\n\ndef generate_merges(vocab, vocab_scores, skip_tokens: Collection[str] | None = None):\n    skip_tokens = set(skip_tokens) if skip_tokens is not None else set()\n    reverse = vocab_scores is not None\n    vocab_scores = dict(vocab_scores) if reverse else vocab\n\n    merges = []","sourceCodeStart":91,"sourceCodeEnd":127,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/convert_slow_tokenizer.py#L91-L127","documentation":"ImportError raised while converting a slow (SentencePiece) tokenizer to fast: parsing the sentencepiece proto requires protobuf, and neither sentencepiece's bundled pb2 nor a standalone google.protobuf install is importable. The message embeds installation instructions and notes a runtime restart may be needed.","triggerScenarios":"Calling AutoTokenizer.from_pretrained(..., use_fast=True) on a sentencepiece checkpoint in an environment without protobuf; or with protobuf installed but only after the process started.","commonSituations":"Minimal Docker images / slurm environments missing optional deps; fresh venvs where only transformers was installed; upgrading protobuf away and leaving a broken state.","solutions":["pip install protobuf (>=4.0.0 preferred; the code picks the right pb2 module per version).","Also ensure sentencepiece is installed for full slow-tokenizer support: pip install sentencepiece protobuf.","Restart the Python runtime/kernel after installing so the new package is importable."],"exampleFix":"// before\nAutoTokenizer.from_pretrained(\"xlm-roberta-base\", use_fast=True)  # ImportError\n\n// after\n# pip install sentencepiece protobuf\nAutoTokenizer.from_pretrained(\"xlm-roberta-base\", use_fast=True)","handlingStrategy":"validation","validationCode":"from transformers.utils import is_protobuf_available\nif not is_protobuf_available():\n    raise RuntimeError(\"Install protobuf before converting sentencepiece tokenizers: pip install protobuf\")","typeGuard":"def can_convert_spm() -> bool:\n    from transformers.utils import is_protobuf_available\n    return is_protobuf_available() or _sentencepiece_pb2_importable()","tryCatchPattern":"try:\n    tok = AutoTokenizer.from_pretrained(repo, use_fast=True)\nexcept ImportError as e:\n    if \"protobuf\" in str(e):\n        tok = AutoTokenizer.from_pretrained(repo, use_fast=False)","preventionTips":["Install optional tokenizer deps up front: pip install 'transformers[tokenizers]' sentencepiece protobuf tiktoken.","Declare optional deps in Dockerfiles/pyproject for any pipeline that touches sentencepiece checkpoints."],"tags":["dependencies","protobuf","tokenizer","conversion"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}