hiyouga/LlamaFactory · error · ValueError

Current model does not support resizing embedding layers.

Error message

Current model does not support resizing embedding layers.

What it means

In the same embedding-resize path, after confirming the model is not quantized, LlamaFactory requires the output projection (lm_head) to be a plain torch.nn.Linear so it can be resized together with the input embedding. If get_output_embeddings() returns None or a tied/non-Linear module while resize is needed, it raises ValueError. This typically happens with weight-tied models or custom heads.

Source

Thrown at src/llamafactory/model/model_utils/embedding.py:303

        import deepspeed  # type: ignore

        params = [model.get_input_embeddings().weight]
        if model.get_output_embeddings() is not None and not model.config.tie_word_embeddings:
            params.append(model.get_output_embeddings().weight)

        context_maybe_zero3 = deepspeed.zero.GatheredParameters(params, modifier_rank=0)
    else:
        context_maybe_zero3 = nullcontext()

    current_embedding_size = get_embedding_vocab_size(model)
    needs_resize = len(tokenizer) > current_embedding_size

    if needs_resize:
        if getattr(model, "quantization_method", None):
            raise ValueError("Cannot resize embedding layers of a quantized model.")

        if not isinstance(model.get_output_embeddings(), torch.nn.Linear):
            raise ValueError("Current model does not support resizing embedding layers.")

        # mean_resizing=False preserves the original embedding distribution exactly.
        # HuggingFace's default mean_resizing=True re-samples new rows from the mean/covariance
        # of existing embeddings, which conflicts with our explicit initialization below.
        model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=64, mean_resizing=False)

    with context_maybe_zero3:
        new_embedding_size = model.get_input_embeddings().weight.size(0)
        num_new_tokens = new_embedding_size - current_embedding_size

        # Resolve the exact rows of the new tokens. This works whether or not a resize was
        # triggered (e.g. tokens added into a model's pre-existing padding zone).
        new_token_ids = _resolve_new_token_ids(new_tokens, tokenizer, new_embedding_size)

        if num_new_tokens <= 0 and not new_token_ids:
            return

        if needs_resize:

View on GitHub (pinned to f28afaf635)

Solutions

1. Use the tokenizer shipped with the model checkpoint so no resize is triggered.
2. If your model ties embeddings, untie/ensure the checkpoint exposes a plain Linear lm_head before training with extra tokens.
3. Patch your custom model class so get_output_embeddings() returns the nn.Linear lm_head.
4. Train without adding tokens and merge new-token semantics into existing vocab instead.

Example fix

# before
tokenizer = AutoTokenizer.from_pretrained("other-model-tokenizer")  # larger vocab

# after
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)  # matching vocab, no resize

out = model.get_output_embeddings()
if len(tokenizer) > model.get_input_embeddings().weight.size(0):
    assert isinstance(out, torch.nn.Linear), (
        "output embeddings must be nn.Linear to resize; untie or fix custom head"
    )

When it happens

Trigger: needs_resize is true (len(tokenizer) > embedding vocab), model is not quantized, but model.get_output_embeddings() is not an nn.Linear — e.g. models with tied embeddings returning None output embeddings, or a custom head class.

Common situations: Fine-tuning models with tie_word_embeddings where the head is not an independent Linear; custom model implementations that return a wrapper module as output embeddings; accidentally pairing a tokenizer from a larger-vocab sibling model.

Related errors

• Cannot resize embedding layers of a quantized model.
• Stop words are required to replace the EOS token.
• YAML config must be a dictionary mapping tokens to descripti
• Cannot resize embedding layers of a quantized model.
• Failed to load tokenizer.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e9b10b475ec7a191. Report an issue: GitHub.