hiyouga/LlamaFactory · error · ValueError

Current model does not support freeze tuning.

Error message

Current model does not support freeze tuning.

What it means

Thrown by the v1 PEFT plugin's freeze-tuning setup when the model config exposes none of the layer-count attributes it probes (num_hidden_layers, num_layers, n_layer). Freeze tuning works by selecting trainable layers by index, so the plugin must know how many decoder layers exist. Architectures that do not report a layer count (some custom or non-transformer models) cannot be freeze-tuned this way.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/peft.py:233

    freeze_trainable_modules = peft_config.freeze_trainable_modules
    freeze_extra_modules = peft_config.freeze_extra_modules
    cast_trainable_params_to_fp32 = peft_config.cast_trainable_params_to_fp32

    if isinstance(freeze_trainable_modules, str):
        freeze_trainable_modules = [module.strip() for module in freeze_trainable_modules.split(",")]

    if isinstance(freeze_extra_modules, str):
        freeze_extra_modules = [module.strip() for module in freeze_extra_modules.split(",")]

    # Get number of layers
    num_layers = (
        getattr(model.config, "num_hidden_layers", None)
        or getattr(model.config, "num_layers", None)
        or getattr(model.config, "n_layer", None)
    )

    if not num_layers:
        raise ValueError("Current model does not support freeze tuning.")

    if freeze_trainable_layers > 0:
        # last n layers
        trainable_layer_ids = range(max(0, num_layers - freeze_trainable_layers), num_layers)
    else:
        # first n layers
        trainable_layer_ids = range(min(-freeze_trainable_layers, num_layers))

    # Identify hidden and non-hidden modules
    hidden_modules = set()
    non_hidden_modules = set()
    for name, _ in model.named_parameters():
        if ".0." in name:
            hidden_modules.add(name.split(".0.")[-1].split(".")[0])
        elif ".1." in name:
            hidden_modules.add(name.split(".1.")[-1].split(".")[0])

        if re.search(r"\.\d+\.", name) is None:

View on GitHub (pinned to f28afaf635)

Solutions

1. Check model.config for a layer-count attribute (print model.config) and confirm the architecture is a stacked decoder/encoder model
2. If the config uses a non-standard attribute name, patch the config or subclass to expose num_hidden_layers before creating the trainer
3. Switch to lora or another peft method that does not need layer indices
4. If you maintain the model code, add num_hidden_layers (or n_layer) to the config class

Example fix

# before
peft_config:
  name: freeze
  freeze_trainable_layers: 8
# model: custom architecture without num_hidden_layers

# after
peft_config:
  name: lora
  lora_rank: 8

def supports_freeze_tuning(model) -> bool:
    cfg = model.config
    return bool(getattr(cfg, "num_hidden_layers", None) or getattr(cfg, "num_layers", None) or getattr(cfg, "n_layer", None))

if not supports_freeze_tuning(model):
    raise SystemExit("model lacks layer-count attr; use lora instead of freeze")

try:
    trainer = create_trainer(cfg)  # freeze setup
except ValueError as e:
    if "freeze tuning" in str(e):
        logger.error("architecture unsupported for freeze; falling back disabled")
    raise

When it happens

Trigger: Setting a freeze peft_config (name 'freeze') on a model whose config lacks all three of num_hidden_layers / num_layers / n_layer, e.g. custom architectures, some Mamba/CNN hybrids, or models loaded with an incomplete config.

Common situations: User switches from a Llama/Qwen checkpoint to an exotic or in-house architecture and reuses the same freeze-tuning YAML; or the model was exported with a minimal config.json that omits the layer-count field.

Related errors

• Module {module_name} not found in hidden modules: {hidden_mo
• Module {module_name} not found in non-hidden modules: {non_h
• Current model does not support freeze tuning.
• When `adapter_name_or_path` is provided for training, only a
• Please specify peft_config to merge and export model.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/bae879f21e090847. Report an issue: GitHub.