hiyouga/LlamaFactory · error · ValueError

`num_layers` {num_layers} should be divisible by `num_expand

Error message

`num_layers` {num_layers} should be divisible by `num_expand` {num_expand}.

What it means

After calling Trainer.__init__, the DPO trainer requires self.accelerator to exist (trainer.py:90). self.accelerator is set by modern transformers Trainers; if the installed transformers is too old (pre-accelerate-integration or a version where the attribute is created later/differently), hasattr fails and the trainer aborts with AttributeError instead of crashing obscurely mid-training.

Source

Thrown at scripts/llama_pro.py:54

def change_name(name: str, old_index: int, new_index: int) -> str:
    return name.replace(f".{old_index:d}.", f".{new_index:d}.")


def block_expansion(
    model_name_or_path: str,
    output_dir: str,
    num_expand: int,
    shard_size: str = "5GB",
    save_safetensors: bool = True,
):
    r"""Perform block expansion for LLaMA, Mistral, Qwen2 or Yi models.

    Usage: python llama_pro.py --model_name_or_path meta-llama/Llama-2-7b-hf --output_dir llama2_pro --num_expand 8
    """
    config: PretrainedConfig = AutoConfig.from_pretrained(model_name_or_path, trust_remote_code=True)
    num_layers = getattr(config, "num_hidden_layers")
    if num_layers % num_expand != 0:
        raise ValueError(f"`num_layers` {num_layers} should be divisible by `num_expand` {num_expand}.")

    setattr(config, "num_hidden_layers", num_layers + num_expand)
    config.save_pretrained(output_dir)

    tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
    tokenizer.save_pretrained(output_dir)

    print(f"Expanding model of {num_layers} layers to {num_layers + num_expand} layers.")
    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path, torch_dtype="auto", device_map="cpu", trust_remote_code=True, low_cpu_mem_usage=True
    )
    assert isinstance(model, PreTrainedModel)  # type hint
    if save_safetensors and getattr(model.config, "tie_word_embeddings", False):
        del model.lm_head  # safetensors does not allow shared weights

    split = num_layers // num_expand
    layer_cnt = 0
    state_dict = model.state_dict()

View on GitHub (pinned to f28afaf635)

Solutions

  1. pip install -U transformers (use the version pinned in LlamaFactory's requirements.txt / pyproject for a known-good pair)
  2. Verify with `python -c "import transformers; print(transformers.__version__)"` after upgrading to confirm no other tool downgraded it

Example fix

# before
transformers 4.3x in env -> AttributeError: Please update `transformers`.

# after
pip install -U transformers
python -c "import transformers; print(transformers.__version__)"
Defensive patterns

Strategy: validation

Validate before calling

import transformers
from packaging.version import parse
v = parse(transformers.__version__)
assert not parse('4.30') <= v < parse('4.37'), (
    f'transformers {v} too old for DPO trainer; pip install -U transformers'
)

Type guard

def transformers_new_enough_for_dpo() -> bool:
    import transformers
    from packaging.version import parse
    return parse(transformers.__version__) >= parse('4.37.0')

Try / catch

try:
    from llamafactory.train.dpo.workflow import run_dpo
    run_dpo(train_args)
except AttributeError as e:
    if 'update `transformers`' in str(e):
        raise SystemExit('pip install -U transformers') from e
    raise

Prevention

When it happens

Trigger: Running DPO/KTO-style training (run_dpo) with an outdated transformers version; Trainer.__init__ completes but never assigns self.accelerator, so the guard raises immediately after super().__init__.

Common situations: Pinning transformers < 4.x-latest for another model while running DPO; environments where another package downgraded transformers; stale conda envs.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f76a2ad99e576a69. Report an issue: GitHub.