hiyouga/LlamaFactory · error · ValueError

Cannot resize embedding layers of a quantized model.

Error message

Cannot resize embedding layers of a quantized model.

What it means

During tokenizer/model alignment in resize_token_embeddings (embedding.py), LlamaFactory must grow the embedding matrix when the tokenizer is larger than the model vocab. Quantized models (GPTQ/AWQ/bitsandbytes/PTQ) store weights in packed/quantized form, so in-place resizing of embedding layers is impossible; the code raises ValueError when model.quantization_method is set.

Source

Thrown at src/llamafactory/model/model_utils/embedding.py:300

        init_special_tokens: Initialization method ('noise_init', 'desc_init', 'desc_init_w_noise')
    """
    if is_deepspeed_zero3_enabled():
        import deepspeed  # type: ignore

        params = [model.get_input_embeddings().weight]
        if model.get_output_embeddings() is not None and not model.config.tie_word_embeddings:
            params.append(model.get_output_embeddings().weight)

        context_maybe_zero3 = deepspeed.zero.GatheredParameters(params, modifier_rank=0)
    else:
        context_maybe_zero3 = nullcontext()

    current_embedding_size = get_embedding_vocab_size(model)
    needs_resize = len(tokenizer) > current_embedding_size

    if needs_resize:
        if getattr(model, "quantization_method", None):
            raise ValueError("Cannot resize embedding layers of a quantized model.")

        if not isinstance(model.get_output_embeddings(), torch.nn.Linear):
            raise ValueError("Current model does not support resizing embedding layers.")

        # mean_resizing=False preserves the original embedding distribution exactly.
        # HuggingFace's default mean_resizing=True re-samples new rows from the mean/covariance
        # of existing embeddings, which conflicts with our explicit initialization below.
        model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=64, mean_resizing=False)

    with context_maybe_zero3:
        new_embedding_size = model.get_input_embeddings().weight.size(0)
        num_new_tokens = new_embedding_size - current_embedding_size

        # Resolve the exact rows of the new tokens. This works whether or not a resize was
        # triggered (e.g. tokens added into a model's pre-existing padding zone).
        new_token_ids = _resolve_new_token_ids(new_tokens, tokenizer, new_embedding_size)

        if num_new_tokens <= 0 and not new_token_ids:

View on GitHub (pinned to f28afaf635)

Solutions

1. Use a tokenizer whose vocabulary fits within the quantized model's embedding size (len(tokenizer) <= embedding rows) — usually the tokenizer shipped with the checkpoint.
2. Fine-tune the non-quantized (or LoRA on fp16/bf16) model when you must add tokens, then quantize after training.
3. Check for accidentally-added tokens: inspect added_tokens and template tokens; ensure cutoff_len/preprocessing does not extend the tokenizer.
4. If tokens were added unintentionally by an enriched tokenizer file, revert to the original tokenizer.

Example fix

# before
model = load_train_model(model_args, finetuning_args)  # quantized + bigger tokenizer

# after: align vocab before quantization, or dequantize first
# 1) train LoRA on the bf16 base, resize embeddings, then export+quantize
# 2) or pass the checkpoint's original tokenizer dir

tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
emb_rows = model.get_input_embeddings().weight.size(0)
if len(tokenizer) > emb_rows:
    assert not getattr(model, "quantization_method", None), (
        "quantized model cannot resize embeddings; align tokenizer vocab first"
    )

When it happens

Trigger: new_tokens / vocab mismatch path: len(tokenizer) > get_embedding_vocab_size(model) while model.quantization_method is truthy — e.g. training a GPTQ or 4-bit bitsandbytes model with a tokenizer that has more tokens (added special tokens, chat template tokens) than the checkpoint's embedding rows.

Common situations: QLoRA fine-tuning of a GPTQ/BNB quantized base model with a different tokenizer file or additional tokens; datasets that force token addition; using quantized checkpoints whose vocab is smaller than the tokenizer.json shipped alongside.

Related errors

• Current model does not support resizing embedding layers.
• Cannot resize embedding layers of a quantized model.
• Stop words are required to replace the EOS token.
• YAML config must be a dictionary mapping tokens to descripti
• Quantization dataset is necessary for exporting.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c6d88e2c65041f30. Report an issue: GitHub.