{"record":{"id":"bb6caf83967facbb","repo":"huggingface/transformers","slug":"tokens-and-scores-need-to-be-passed-for-a-llama-to","errorCode":null,"errorMessage":"tokens and scores need to be passed for a LLaMa tokenizer without merges to be instantiated.","messagePattern":"tokens and scores need to be passed for a LLaMa tokenizer without merges to be instantiated\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/ggml.py","lineNumber":413,"sourceCode":"    elif data_type in [6, 12]:\n        _value = float(_value[0])\n    elif data_type == 7:\n        _value = bool(_value[0])\n    elif data_type == 8:\n        _value = array(\"B\", list(_value)).tobytes().decode()\n    elif data_type == 9:\n        _value = _gguf_parse_value(_value, array_data_type)\n    return _value\n\n\nclass GGUFTokenizerSkeleton:\n    def __init__(self, dict_):\n        for k, v in dict_.items():\n            setattr(self, k, v)\n\n        if not hasattr(self, \"merges\"):\n            if not hasattr(self, \"tokens\") or not hasattr(self, \"scores\"):\n                raise ValueError(\n                    \"tokens and scores need to be passed for a LLaMa tokenizer without merges to be instantiated.\"\n                )\n            tokens = self.tokens\n            scores = self.scores\n            vocab = {t: scores[i] for i, t in enumerate(tokens)}\n\n            logger.warning(\"Merges were not in checkpoint, building merges on the fly.\")\n            merges = []\n            for merge, piece_score in tqdm(vocab.items()):\n                local = []\n                for index in range(1, len(merge)):\n                    piece_l, piece_r = merge[:index], merge[index:]\n                    if piece_l in tokens and piece_r in tokens:\n                        local.append((piece_l, piece_r, piece_score))\n                local = sorted(local, key=lambda x: (vocab[x[0]], vocab[x[1]]), reverse=True)\n                merges.extend(local)\n            merges = sorted(merges, key=lambda val: val[2], reverse=True)\n            merges = [(val[0], val[1]) for val in merges]","sourceCodeStart":395,"sourceCodeEnd":431,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/ggml.py#L395-L431","documentation":"GGUFTokenizerSkeleton builds a tokenizer from GGUF tokenizer metadata. If the metadata contains no merges, it falls back to building a LLaMa-style (SPM) tokenizer on the fly — but that fallback requires the token list and per-token scores. If either tokens or scores is absent from the GGUF metadata, the skeleton cannot construct a vocabulary at all and raises ValueError stating exactly what is missing.","triggerScenarios":"Loading a GGUF model whose tokenizer.ggml metadata lacks tokenizer.merges AND lacks either tokenizer.ggml.tokens or tokenizer.ggml.scores (or the reader failed to surface them) — then instantiating the tokenizer via GGUFTokenizerSkeleton.","commonSituations":"Nonstandard GGUF conversions that embed a tokenizer without scores (e.g. BPE-only files missing both merges and score fields); truncated metadata after a bad upload; experimental GGUF producers.","solutions":["Re-convert the model with a converter that writes complete tokenizer metadata (tokens + scores, or merges for BPE)","Load the tokenizer separately from its HF repo (AutoTokenizer.from_pretrained on the original model) instead of relying on GGUF-embedded metadata","Inspect the file with gguf-dump to confirm which tokenizer.* fields exist"],"exampleFix":"# before: GGUF lacks merges and scores\nmodel = AutoModelForCausalLM.from_pretrained(\"model.gguf\")  # ValueError at tokenizer build\n\n# after: load tokenizer from the original HF repo\ntokenizer = AutoTokenizer.from_pretrained(\"original-model-repo\")\nmodel = AutoModelForCausalLM.from_pretrained(\"model.gguf\")","handlingStrategy":"fallback","validationCode":"from gguf import GGUFReader\nr = GGUFReader(\"model.gguf\")\nfields = {part if isinstance(part, str) else part.decode() for field in r.fields.values() for part in [field.name]}\nhas_tokenizer_meta = (\"tokenizer.ggml.tokens\" in fields and \"tokenizer.ggml.scores\" in fields) or \"tokenizer.ggml.merges\" in fields","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Prefer loading the tokenizer from the original HF repo rather than GGUF metadata","Check that conversions include tokenizer.ggml.tokens and tokenizer.ggml.scores for SPM models"],"tags":["gguf","tokenizer","metadata-parsing","llama-spm"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}