huggingface/transformers · error · ValueError

tokens and scores need to be passed for a LLaMa tokenizer wi

Error message

tokens and scores need to be passed for a LLaMa tokenizer without merges to be instantiated.

What it means

GGUFTokenizerSkeleton builds a tokenizer from GGUF tokenizer metadata. If the metadata contains no merges, it falls back to building a LLaMa-style (SPM) tokenizer on the fly — but that fallback requires the token list and per-token scores. If either tokens or scores is absent from the GGUF metadata, the skeleton cannot construct a vocabulary at all and raises ValueError stating exactly what is missing.

Source

Thrown at src/transformers/integrations/ggml.py:413

    elif data_type in [6, 12]:
        _value = float(_value[0])
    elif data_type == 7:
        _value = bool(_value[0])
    elif data_type == 8:
        _value = array("B", list(_value)).tobytes().decode()
    elif data_type == 9:
        _value = _gguf_parse_value(_value, array_data_type)
    return _value


class GGUFTokenizerSkeleton:
    def __init__(self, dict_):
        for k, v in dict_.items():
            setattr(self, k, v)

        if not hasattr(self, "merges"):
            if not hasattr(self, "tokens") or not hasattr(self, "scores"):
                raise ValueError(
                    "tokens and scores need to be passed for a LLaMa tokenizer without merges to be instantiated."
                )
            tokens = self.tokens
            scores = self.scores
            vocab = {t: scores[i] for i, t in enumerate(tokens)}

            logger.warning("Merges were not in checkpoint, building merges on the fly.")
            merges = []
            for merge, piece_score in tqdm(vocab.items()):
                local = []
                for index in range(1, len(merge)):
                    piece_l, piece_r = merge[:index], merge[index:]
                    if piece_l in tokens and piece_r in tokens:
                        local.append((piece_l, piece_r, piece_score))
                local = sorted(local, key=lambda x: (vocab[x[0]], vocab[x[1]]), reverse=True)
                merges.extend(local)
            merges = sorted(merges, key=lambda val: val[2], reverse=True)
            merges = [(val[0], val[1]) for val in merges]

View on GitHub (pinned to a597f97485)

Solutions

1. Re-convert the model with a converter that writes complete tokenizer metadata (tokens + scores, or merges for BPE)
2. Load the tokenizer separately from its HF repo (AutoTokenizer.from_pretrained on the original model) instead of relying on GGUF-embedded metadata
3. Inspect the file with gguf-dump to confirm which tokenizer.* fields exist

Example fix

# before: GGUF lacks merges and scores
model = AutoModelForCausalLM.from_pretrained("model.gguf")  # ValueError at tokenizer build

# after: load tokenizer from the original HF repo
tokenizer = AutoTokenizer.from_pretrained("original-model-repo")
model = AutoModelForCausalLM.from_pretrained("model.gguf")

from gguf import GGUFReader
r = GGUFReader("model.gguf")
fields = {part if isinstance(part, str) else part.decode() for field in r.fields.values() for part in [field.name]}
has_tokenizer_meta = ("tokenizer.ggml.tokens" in fields and "tokenizer.ggml.scores" in fields) or "tokenizer.ggml.merges" in fields

When it happens

Trigger: Loading a GGUF model whose tokenizer.ggml metadata lacks tokenizer.merges AND lacks either tokenizer.ggml.tokens or tokenizer.ggml.scores (or the reader failed to surface them) — then instantiating the tokenizer via GGUFTokenizerSkeleton.

Common situations: Nonstandard GGUF conversions that embed a tokenizer without scores (e.g. BPE-only files missing both merges and score fields); truncated metadata after a bad upload; experimental GGUF producers.

Related errors

• Received multiple types, therefore expected the first type t
• {error_message} requires the protobuf library but it was not
• You're trying to run a `Unigram` model but you're file was t
• `tiktoken` is required to read a `tiktoken` file. Install it
• Converting from SentencePiece and Tiktoken failed, if a conv

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/bb6caf83967facbb. Report an issue: GitHub.