huggingface/transformers · error · ImportError

{error_message} requires the protobuf library but it was not

Error message

{error_message} requires the protobuf library but it was not found in your environment. Check out the instructions on the installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones that match your environment. Please note that you may need to restart your runtime after installation.

What it means

ImportError raised while converting a slow (SentencePiece) tokenizer to fast: parsing the sentencepiece proto requires protobuf, and neither sentencepiece's bundled pb2 nor a standalone google.protobuf install is importable. The message embeds installation instructions and notes a runtime restart may be needed.

Source

Thrown at src/transformers/convert_slow_tokenizer.py:109

    "sl_SI",
]


def import_protobuf(error_message=""):
    if is_sentencepiece_available():
        from sentencepiece import sentencepiece_model_pb2

        return sentencepiece_model_pb2
    if is_protobuf_available():
        import google.protobuf

        if version.parse(google.protobuf.__version__) < version.parse("4.0.0"):
            from transformers.utils import sentencepiece_model_pb2
        else:
            from transformers.utils import sentencepiece_model_pb2_new as sentencepiece_model_pb2
        return sentencepiece_model_pb2
    else:
        raise ImportError(PROTOBUF_IMPORT_ERROR.format(error_message))


def _get_prepend_scheme(add_prefix_space: bool, original_tokenizer) -> str:
    if add_prefix_space:
        prepend_scheme = "always"
        if not getattr(original_tokenizer, "legacy", True):
            prepend_scheme = "first"
    else:
        prepend_scheme = "never"
    return prepend_scheme


def generate_merges(vocab, vocab_scores, skip_tokens: Collection[str] | None = None):
    skip_tokens = set(skip_tokens) if skip_tokens is not None else set()
    reverse = vocab_scores is not None
    vocab_scores = dict(vocab_scores) if reverse else vocab

    merges = []

View on GitHub (pinned to a597f97485)

Solutions

1. pip install protobuf (>=4.0.0 preferred; the code picks the right pb2 module per version).
2. Also ensure sentencepiece is installed for full slow-tokenizer support: pip install sentencepiece protobuf.
3. Restart the Python runtime/kernel after installing so the new package is importable.

Example fix

// before
AutoTokenizer.from_pretrained("xlm-roberta-base", use_fast=True)  # ImportError

// after
# pip install sentencepiece protobuf
AutoTokenizer.from_pretrained("xlm-roberta-base", use_fast=True)

from transformers.utils import is_protobuf_available
if not is_protobuf_available():
    raise RuntimeError("Install protobuf before converting sentencepiece tokenizers: pip install protobuf")

def can_convert_spm() -> bool:
    from transformers.utils import is_protobuf_available
    return is_protobuf_available() or _sentencepiece_pb2_importable()

try:
    tok = AutoTokenizer.from_pretrained(repo, use_fast=True)
except ImportError as e:
    if "protobuf" in str(e):
        tok = AutoTokenizer.from_pretrained(repo, use_fast=False)

When it happens

Trigger: Calling AutoTokenizer.from_pretrained(..., use_fast=True) on a sentencepiece checkpoint in an environment without protobuf; or with protobuf installed but only after the process started.

Common situations: Minimal Docker images / slurm environments missing optional deps; fresh venvs where only transformers was installed; upgrading protobuf away and leaving a broken state.

Related errors

• `tiktoken` is required to read a `tiktoken` file. Install it
• You're trying to run a `Unigram` model but you're file was t
• Converting from SentencePiece and Tiktoken failed, if a conv
• Unrecognized tokenizer name, should be one of {list(TOKENIZE
• You need to install optimum-quanto in order to use KV cache

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/af014127abd310dd. Report an issue: GitHub.