hiyouga/LlamaFactory · error · ValueError

Module {module_name} not found in non-hidden modules: {non_h

Error message

Module {module_name} not found in non-hidden modules: {non_hidden_modules}

What it means

Freeze tuning validates freeze_extra_modules against non_hidden_modules, the set of parameter-parent names that appear OUTSIDE the numbered layer blocks (e.g. 'embed_tokens', 'norm', 'lm_head'). If an extra module name is not found there, no parameter would match it, so the plugin rejects the config. The name must be an existing top-level (non-layer) submodule of the model.

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/peft.py:272

    # Build list of trainable layer patterns
    trainable_layers = []
    for module_name in freeze_trainable_modules:
        if module_name == "all":
            for idx in trainable_layer_ids:
                trainable_layers.append(f".{idx:d}.")
        elif module_name in hidden_modules:
            for idx in trainable_layer_ids:
                trainable_layers.append(f".{idx:d}.{module_name}")
        else:
            raise ValueError(f"Module {module_name} not found in hidden modules: {hidden_modules}")

    # Add extra modules
    if freeze_extra_modules:
        for module_name in freeze_extra_modules:
            if module_name in non_hidden_modules:
                trainable_layers.append(module_name)
            else:
                raise ValueError(f"Module {module_name} not found in non-hidden modules: {non_hidden_modules}")

    # TODO
    # Multi-modal special handling

    # Set requires_grad
    forbidden_modules = {"quant_state", "quantization_weight", "qweight", "qzeros", "scales"}
    for name, param in model.named_parameters():
        if any(trainable_layer in name for trainable_layer in trainable_layers) and not any(
            forbidden_module in name for forbidden_module in forbidden_modules
        ):
            param.requires_grad_(True)
            if cast_trainable_params_to_fp32:
                param.data = param.data.to(torch.float32)  # Cast to fp32 for stability
        else:
            param.requires_grad_(False)

    logger.info_rank0(f"Set trainable layers: {trainable_layers}")

View on GitHub (pinned to f28afaf635)

Solutions

1. Inspect non-hidden parameter names: print({n.split('.')[-2] for n,_ in model.named_parameters() if not any(f'.{i}.' in n for i in range(100))})
2. Use the exact names, typically 'embed_tokens' and 'norm' (and 'lm_head' if untied)
3. Remove modules that live inside layers from freeze_extra_modules and put them in freeze_trainable_modules

Example fix

# before
freeze_extra_modules: ["embedding", "head"]

# after
freeze_extra_modules: ["embed_tokens", "norm", "lm_head"]

import re
non_hidden = {n.split(".")[-2] for n, _ in model.named_parameters() if not re.search(r"\.\d+\.", n)}
missing = [m for m in freeze_extra_modules if m not in non_hidden]
assert not missing, f"extra modules not found: {missing}; valid: {sorted(non_hidden)}"

When it happens

Trigger: freeze_extra_modules lists a name like 'embeddings' or 'head' that does not occur as the second-to-last component of any parameter outside the decoder layers; or the name exists only inside layers (belongs in freeze_trainable_modules instead).

Common situations: Users try to unfreeze embeddings/head with guessed names ('embedding', 'output') instead of the model's actual names (embed_tokens, lm_head); mixing up freeze_trainable_modules and freeze_extra_modules semantics.

Related errors

• Module {module_name} not found in hidden modules: {hidden_mo
• Current model does not support freeze tuning.
• Module {} is not found, please choose from {}
• When `adapter_name_or_path` is provided for training, only a
• Please specify peft_config to merge and export model.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/12e102722cb1ddc4. Report an issue: GitHub.