{"record":{"id":"d21bdf5a434ba69d","repo":"hiyouga/LlamaFactory","slug":"per-layer-galore-does-not-support-gradient-accumul","errorCode":null,"errorMessage":"Per-layer GaLore does not support gradient accumulation.","messagePattern":"Per-layer GaLore does not support gradient accumulation\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/train/trainer_utils.py","lineNumber":251,"sourceCode":"                    decay_params.append(param)\n                else:\n                    nodecay_params.append(param)\n\n    _, optim_kwargs = Trainer.get_optimizer_cls_and_kwargs(training_args)\n\n    if training_args.optim == \"adamw_torch\":\n        optim_class = GaLoreAdamW\n    elif training_args.optim in [\"adamw_bnb_8bit\", \"adamw_8bit\", \"paged_adamw_8bit\"]:\n        optim_class = GaLoreAdamW8bit\n    elif training_args.optim == \"adafactor\":\n        optim_class = GaLoreAdafactor\n    else:\n        raise NotImplementedError(f\"Unknown optim: {training_args.optim}.\")\n\n    if finetuning_args.galore_layerwise:\n        logger.warning_rank0(\"The displayed gradient norm will be all zeros in layerwise GaLore.\")\n        if training_args.gradient_accumulation_steps != 1:\n            raise ValueError(\"Per-layer GaLore does not support gradient accumulation.\")\n\n        optimizer_dict: dict[torch.Tensor, torch.optim.Optimizer] = {}\n        for param in nodecay_params:\n            param_groups = [dict(params=[param], weight_decay=0.0)]\n            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)\n        for param in decay_params:\n            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay)]\n            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)\n        for param in galore_params:  # galore params have weight decay\n            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay, **galore_kwargs)]\n            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)\n\n        def optimizer_hook(param: \"torch.nn.Parameter\"):\n            if param.grad is not None:\n                optimizer_dict[param].step()\n                optimizer_dict[param].zero_grad()\n\n        for param in trainable_params:","sourceCodeStart":233,"sourceCodeEnd":269,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/train/trainer_utils.py#L233-L269","documentation":"Layerwise GaLore (src/llamafactory/train/trainer_utils.py:251) swaps in a per-parameter optimizer that steps immediately per layer; gradient accumulation across micro-batches is incompatible with that scheme, so `gradient_accumulation_steps != 1` is rejected. The code even logs first that displayed gradient norms will be zero in layerwise mode.","triggerScenarios":"Config with `use_galore: true`, `galore_layerwise: true`, and `gradient_accumulation_steps: > 1` (2, 4, 8 are common defaults for large effective batch sizes).","commonSituations":"Users simulating a large batch with accumulation on limited VRAM while also enabling layerwise GaLore to save memory; template configs that ship with gradient_accumulation_steps: 8.","solutions":["Set `gradient_accumulation_steps: 1` when using galore_layerwise.","To keep the effective batch size, increase `per_device_train_batch_size` instead, or enable packing.","If accumulation is mandatory, disable `galore_layerwise` and use standard (projected) GaLore."],"exampleFix":"# before (yaml)\nuse_galore: true\ngalore_layerwise: true\ngradient_accumulation_steps: 8\n\n# after\nuse_galore: true\ngalore_layerwise: true\ngradient_accumulation_steps: 1\nper_device_train_batch_size: 8","handlingStrategy":"validation","validationCode":"if use_galore and galore_layerwise:\n    assert gradient_accumulation_steps == 1, \"layerwise GaLore requires gradient_accumulation_steps=1\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check incompatible-flag pairs (layerwise + accumulation) in a config preflight script.","Adjust batch size instead of accumulation when using layerwise modes."],"tags":["galore","layerwise","gradient-accumulation","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}