hiyouga/LlamaFactory · error · NotImplementedError

Unknown optim: {training_args.optim}.

Error message

Unknown optim: {training_args.optim}.

What it means

The GaLore optimizer factory (src/llamafactory/train/trainer_utils.py:246) only maps three optimizer choices onto GaLore variants: `adamw_torch` -> GaLoreAdamW, the bitsandbytes 8-bit family (`adamw_bnb_8bit`, `adamw_8bit`, `paged_adamw_8bit`) -> GaLoreAdamW8bit, and `adafactor` -> GaLoreAdafactor. Any other `optim` value raises NotImplementedError because no GaLore equivalent exists for it.

Source

Thrown at src/llamafactory/train/trainer_utils.py:246

    for name, param in model.named_parameters():
        if param.requires_grad:
            trainable_params.append(param)
            if id(param) not in id_galore_params:
                if name in decay_param_names:
                    decay_params.append(param)
                else:
                    nodecay_params.append(param)

    _, optim_kwargs = Trainer.get_optimizer_cls_and_kwargs(training_args)

    if training_args.optim == "adamw_torch":
        optim_class = GaLoreAdamW
    elif training_args.optim in ["adamw_bnb_8bit", "adamw_8bit", "paged_adamw_8bit"]:
        optim_class = GaLoreAdamW8bit
    elif training_args.optim == "adafactor":
        optim_class = GaLoreAdafactor
    else:
        raise NotImplementedError(f"Unknown optim: {training_args.optim}.")

    if finetuning_args.galore_layerwise:
        logger.warning_rank0("The displayed gradient norm will be all zeros in layerwise GaLore.")
        if training_args.gradient_accumulation_steps != 1:
            raise ValueError("Per-layer GaLore does not support gradient accumulation.")

        optimizer_dict: dict[torch.Tensor, torch.optim.Optimizer] = {}
        for param in nodecay_params:
            param_groups = [dict(params=[param], weight_decay=0.0)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)
        for param in decay_params:
            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)
        for param in galore_params:  # galore params have weight decay
            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay, **galore_kwargs)]
            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)

        def optimizer_hook(param: "torch.nn.Parameter"):

View on GitHub (pinned to f28afaf635)

Solutions

1. Set `optim: adamw_torch` in the training YAML when using GaLore.
2. For memory-constrained runs use `optim: adamw_bnb_8bit` (requires bitsandbytes installed).
3. If you do not need GaLore, remove `use_galore: true` and keep your preferred optimizer.
4. For adafactor-style training use `optim: adafactor`.

Example fix

# before (yaml)
use_galore: true
optim: adamw_torch_fused

# after
use_galore: true
optim: adamw_torch

GALORE_OPTIMS = {"adamw_torch", "adamw_bnb_8bit", "adamw_8bit", "paged_adamw_8bit", "adafactor"}
assert optim in GALORE_OPTIMS or not use_galore

When it happens

Trigger: Config with `use_galore: true` plus an unsupported optimizer such as `adamw_torch_fused`, `adamw_anyprecision`, `lion`, or `adahessian` in TrainingArguments.

Common situations: Users copying a tuned optimizer setting from a non-GaLore run into a GaLore config, or defaults from a newer transformers version introducing optim names this code predates.

Related errors

• Cannot use LoRA with GaLore, APOLLO or BAdam together.
• Cannot use GaLore, APOLLO or BAdam together.
• The length of packed example should be identical to the cuto
• `loraplus_lr_ratio` is only valid for LoRA training.
• `predict_with_generate` cannot be set as True except SFT.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/3f69955c12070d82. Report an issue: GitHub.