{"record":{"id":"c6d88e2c65041f30","repo":"hiyouga/LlamaFactory","slug":"cannot-resize-embedding-layers-of-a-quantized-mode-c6d88e","errorCode":null,"errorMessage":"Cannot resize embedding layers of a quantized model.","messagePattern":"Cannot resize embedding layers of a quantized model\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/embedding.py","lineNumber":300,"sourceCode":"        init_special_tokens: Initialization method ('noise_init', 'desc_init', 'desc_init_w_noise')\n    \"\"\"\n    if is_deepspeed_zero3_enabled():\n        import deepspeed  # type: ignore\n\n        params = [model.get_input_embeddings().weight]\n        if model.get_output_embeddings() is not None and not model.config.tie_word_embeddings:\n            params.append(model.get_output_embeddings().weight)\n\n        context_maybe_zero3 = deepspeed.zero.GatheredParameters(params, modifier_rank=0)\n    else:\n        context_maybe_zero3 = nullcontext()\n\n    current_embedding_size = get_embedding_vocab_size(model)\n    needs_resize = len(tokenizer) > current_embedding_size\n\n    if needs_resize:\n        if getattr(model, \"quantization_method\", None):\n            raise ValueError(\"Cannot resize embedding layers of a quantized model.\")\n\n        if not isinstance(model.get_output_embeddings(), torch.nn.Linear):\n            raise ValueError(\"Current model does not support resizing embedding layers.\")\n\n        # mean_resizing=False preserves the original embedding distribution exactly.\n        # HuggingFace's default mean_resizing=True re-samples new rows from the mean/covariance\n        # of existing embeddings, which conflicts with our explicit initialization below.\n        model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=64, mean_resizing=False)\n\n    with context_maybe_zero3:\n        new_embedding_size = model.get_input_embeddings().weight.size(0)\n        num_new_tokens = new_embedding_size - current_embedding_size\n\n        # Resolve the exact rows of the new tokens. This works whether or not a resize was\n        # triggered (e.g. tokens added into a model's pre-existing padding zone).\n        new_token_ids = _resolve_new_token_ids(new_tokens, tokenizer, new_embedding_size)\n\n        if num_new_tokens <= 0 and not new_token_ids:","sourceCodeStart":282,"sourceCodeEnd":318,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/embedding.py#L282-L318","documentation":"During tokenizer/model alignment in resize_token_embeddings (embedding.py), LlamaFactory must grow the embedding matrix when the tokenizer is larger than the model vocab. Quantized models (GPTQ/AWQ/bitsandbytes/PTQ) store weights in packed/quantized form, so in-place resizing of embedding layers is impossible; the code raises ValueError when model.quantization_method is set.","triggerScenarios":"new_tokens / vocab mismatch path: len(tokenizer) > get_embedding_vocab_size(model) while model.quantization_method is truthy — e.g. training a GPTQ or 4-bit bitsandbytes model with a tokenizer that has more tokens (added special tokens, chat template tokens) than the checkpoint's embedding rows.","commonSituations":"QLoRA fine-tuning of a GPTQ/BNB quantized base model with a different tokenizer file or additional tokens; datasets that force token addition; using quantized checkpoints whose vocab is smaller than the tokenizer.json shipped alongside.","solutions":["Use a tokenizer whose vocabulary fits within the quantized model's embedding size (len(tokenizer) <= embedding rows) — usually the tokenizer shipped with the checkpoint.","Fine-tune the non-quantized (or LoRA on fp16/bf16) model when you must add tokens, then quantize after training.","Check for accidentally-added tokens: inspect added_tokens and template tokens; ensure cutoff_len/preprocessing does not extend the tokenizer.","If tokens were added unintentionally by an enriched tokenizer file, revert to the original tokenizer."],"exampleFix":"# before\nmodel = load_train_model(model_args, finetuning_args)  # quantized + bigger tokenizer\n\n# after: align vocab before quantization, or dequantize first\n# 1) train LoRA on the bf16 base, resize embeddings, then export+quantize\n# 2) or pass the checkpoint's original tokenizer dir","handlingStrategy":"validation","validationCode":"tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)\nemb_rows = model.get_input_embeddings().weight.size(0)\nif len(tokenizer) > emb_rows:\n    assert not getattr(model, \"quantization_method\", None), (\n        \"quantized model cannot resize embeddings; align tokenizer vocab first\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always load the tokenizer shipped with the quantized checkpoint.","Prefer training on the unquantized base and quantizing after export when tokens must be added."],"tags":["quantization","tokenizer","embedding","resize"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}