{"record":{"id":"d5a8115372ce999f","repo":"hiyouga/LlamaFactory","slug":"per-layer-apollo-does-not-support-gradient-accumul","errorCode":null,"errorMessage":"Per-layer APOLLO does not support gradient accumulation.","messagePattern":"Per-layer APOLLO does not support gradient accumulation\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/train/trainer_utils.py","lineNumber":338,"sourceCode":"        if param.requires_grad:\n            trainable_params.append(param)\n            if id(param) not in id_apollo_params:\n                if name in decay_param_names:\n                    decay_params.append(param)\n                else:\n                    nodecay_params.append(param)\n\n    _, optim_kwargs = Trainer.get_optimizer_cls_and_kwargs(training_args)\n\n    if training_args.optim == \"adamw_torch\":\n        optim_class = APOLLOAdamW\n    else:\n        raise NotImplementedError(f\"Unknown optim: {training_args.optim}.\")\n\n    if finetuning_args.apollo_layerwise:\n        logger.warning_rank0(\"The displayed gradient norm will be all zeros in layerwise APOLLO.\")\n        if training_args.gradient_accumulation_steps != 1:\n            raise ValueError(\"Per-layer APOLLO does not support gradient accumulation.\")\n\n        optimizer_dict: dict[torch.Tensor, torch.optim.Optimizer] = {}\n        for param in nodecay_params:\n            param_groups = [dict(params=[param], weight_decay=0.0)]\n            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)\n        for param in decay_params:\n            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay)]\n            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)\n        for param in apollo_params:  # apollo params have weight decay\n            param_groups = [dict(params=[param], weight_decay=training_args.weight_decay, **apollo_kwargs)]\n            optimizer_dict[param] = optim_class(param_groups, **optim_kwargs)\n\n        def optimizer_hook(param: \"torch.nn.Parameter\"):\n            if param.grad is not None:\n                optimizer_dict[param].step()\n                optimizer_dict[param].zero_grad()\n\n        for param in trainable_params:","sourceCodeStart":320,"sourceCodeEnd":356,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/train/trainer_utils.py#L320-L356","documentation":"Layerwise APOLLO (src/llamafactory/train/trainer_utils.py:338) has the same per-layer immediate-step design as layerwise GaLore, so gradient accumulation is rejected: `apollo_layerwise: true` with `gradient_accumulation_steps != 1` raises ValueError. A prior warning notes displayed grad norms will be zero.","triggerScenarios":"Config with `use_apollo: true`, `apollo_layerwise: true`, and `gradient_accumulation_steps: 2/4/8/...`.","commonSituations":"Memory-optimized large-model runs combining layerwise APOLLO with accumulation; default YAML templates that already set accumulation > 1.","solutions":["Set `gradient_accumulation_steps: 1` with apollo_layerwise.","Compensate effective batch size via per_device_train_batch_size or packing.","Turn off `apollo_layerwise` (keep use_apollo) if accumulation is required."],"exampleFix":"# before (yaml)\nuse_apollo: true\napollo_layerwise: true\ngradient_accumulation_steps: 4\n\n# after\nuse_apollo: true\napollo_layerwise: true\ngradient_accumulation_steps: 1","handlingStrategy":"validation","validationCode":"if use_apollo and apollo_layerwise:\n    assert gradient_accumulation_steps == 1, \"layerwise APOLLO requires gradient_accumulation_steps=1\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Preflight-check layerwise + accumulation pairs for both GaLore and APOLLO.","Scale effective batch via batch size or packing, not accumulation, in layerwise runs."],"tags":["apollo","layerwise","gradient-accumulation","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}