{"record":{"id":"e9b10b475ec7a191","repo":"hiyouga/LlamaFactory","slug":"current-model-does-not-support-resizing-embedding","errorCode":null,"errorMessage":"Current model does not support resizing embedding layers.","messagePattern":"Current model does not support resizing embedding layers\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/model/model_utils/embedding.py","lineNumber":303,"sourceCode":"        import deepspeed  # type: ignore\n\n        params = [model.get_input_embeddings().weight]\n        if model.get_output_embeddings() is not None and not model.config.tie_word_embeddings:\n            params.append(model.get_output_embeddings().weight)\n\n        context_maybe_zero3 = deepspeed.zero.GatheredParameters(params, modifier_rank=0)\n    else:\n        context_maybe_zero3 = nullcontext()\n\n    current_embedding_size = get_embedding_vocab_size(model)\n    needs_resize = len(tokenizer) > current_embedding_size\n\n    if needs_resize:\n        if getattr(model, \"quantization_method\", None):\n            raise ValueError(\"Cannot resize embedding layers of a quantized model.\")\n\n        if not isinstance(model.get_output_embeddings(), torch.nn.Linear):\n            raise ValueError(\"Current model does not support resizing embedding layers.\")\n\n        # mean_resizing=False preserves the original embedding distribution exactly.\n        # HuggingFace's default mean_resizing=True re-samples new rows from the mean/covariance\n        # of existing embeddings, which conflicts with our explicit initialization below.\n        model.resize_token_embeddings(len(tokenizer), pad_to_multiple_of=64, mean_resizing=False)\n\n    with context_maybe_zero3:\n        new_embedding_size = model.get_input_embeddings().weight.size(0)\n        num_new_tokens = new_embedding_size - current_embedding_size\n\n        # Resolve the exact rows of the new tokens. This works whether or not a resize was\n        # triggered (e.g. tokens added into a model's pre-existing padding zone).\n        new_token_ids = _resolve_new_token_ids(new_tokens, tokenizer, new_embedding_size)\n\n        if num_new_tokens <= 0 and not new_token_ids:\n            return\n\n        if needs_resize:","sourceCodeStart":285,"sourceCodeEnd":321,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/model/model_utils/embedding.py#L285-L321","documentation":"In the same embedding-resize path, after confirming the model is not quantized, LlamaFactory requires the output projection (lm_head) to be a plain torch.nn.Linear so it can be resized together with the input embedding. If get_output_embeddings() returns None or a tied/non-Linear module while resize is needed, it raises ValueError. This typically happens with weight-tied models or custom heads.","triggerScenarios":"needs_resize is true (len(tokenizer) > embedding vocab), model is not quantized, but model.get_output_embeddings() is not an nn.Linear — e.g. models with tied embeddings returning None output embeddings, or a custom head class.","commonSituations":"Fine-tuning models with tie_word_embeddings where the head is not an independent Linear; custom model implementations that return a wrapper module as output embeddings; accidentally pairing a tokenizer from a larger-vocab sibling model.","solutions":["Use the tokenizer shipped with the model checkpoint so no resize is triggered.","If your model ties embeddings, untie/ensure the checkpoint exposes a plain Linear lm_head before training with extra tokens.","Patch your custom model class so get_output_embeddings() returns the nn.Linear lm_head.","Train without adding tokens and merge new-token semantics into existing vocab instead."],"exampleFix":"# before\ntokenizer = AutoTokenizer.from_pretrained(\"other-model-tokenizer\")  # larger vocab\n\n# after\ntokenizer = AutoTokenizer.from_pretrained(model_name_or_path)  # matching vocab, no resize","handlingStrategy":"validation","validationCode":"out = model.get_output_embeddings()\nif len(tokenizer) > model.get_input_embeddings().weight.size(0):\n    assert isinstance(out, torch.nn.Linear), (\n        \"output embeddings must be nn.Linear to resize; untie or fix custom head\"\n    )","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Avoid mixing tokenizers from sibling models with different vocab sizes.","For custom architectures, ensure get_output_embeddings() returns the plain Linear lm_head."],"tags":["embedding","tokenizer","lm-head","resize"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}