{"record":{"id":"fab7cadcc16ffc67","repo":"huggingface/pytorch-image-models","slug":"requires-grad-is-not-supported-for-step-in-dif","errorCode":null,"errorMessage":"`requires_grad` is not supported for `step` in differentiable mode","messagePattern":"`requires_grad` is not supported for `step` in differentiable mode","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/adopt.py","lineNumber":178,"sourceCode":"            if len(state) == 0:\n                # note(crcrpar): [special device hosting for step]\n                # Deliberately host `step` on CPU if both capturable and fused are off.\n                # This is because kernel launches are costly on CUDA and XLA.\n                state[\"step\"] = (\n                    torch.zeros((), dtype=_get_scalar_dtype(), device=p.grad.device)\n                    if group[\"capturable\"]\n                    else torch.tensor(0.0, dtype=_get_scalar_dtype())\n                )\n                # Exponential moving average of gradient values\n                state[\"exp_avg\"] = torch.zeros_like(p.grad, memory_format=torch.preserve_format)\n                # Exponential moving average of squared gradient values\n                state[\"exp_avg_sq\"] = torch.zeros_like(p.grad, memory_format=torch.preserve_format)\n\n            exp_avgs.append(state[\"exp_avg\"])\n            exp_avg_sqs.append(state[\"exp_avg_sq\"])\n\n            if group[\"differentiable\"] and state[\"step\"].requires_grad:\n                raise RuntimeError(\"`requires_grad` is not supported for `step` in differentiable mode\")\n\n            # Foreach without capturable does not support a tensor lr\n            if group[\"foreach\"] and torch.is_tensor(group[\"lr\"]) and not group[\"capturable\"]:\n                raise RuntimeError(\"lr as a Tensor is not supported for capturable=False and foreach=True\")\n\n            state_steps.append(state[\"step\"])\n        return has_complex\n\n    #@_use_grad_for_differentiable  # FIXME internal context mgr, can't use\n    @torch.no_grad()\n    def step(self, closure=None):\n        \"\"\"Perform a single optimization step.\n\n        Args:\n            closure (Callable, optional): A closure that reevaluates the model\n                and returns the loss.\n        \"\"\"\n        if hasattr(self, '_accelerator_graph_capture_health_check'):","sourceCodeStart":160,"sourceCodeEnd":196,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/adopt.py#L160-L196","documentation":"In differentiable optimization mode (differentiable=True), ADOPT computes gradients through the optimizer step; if the per-parameter step counter tensor itself has requires_grad=True the higher-order autograd graph is ill-defined, so the optimizer refuses.","triggerScenarios":"Setting differentiable=True in timm.optim.Adopt and manually creating state['step'] tensors with requires_grad=True (e.g. when reimplementing state init), then calling step().","commonSituations":"Hyperparameter-optimization code (meta-gradients through the optimizer) that marks every state tensor as differentiable, including step counters; porting a differentiable-optimizer implementation from torch.optim.Adam.","solutions":["Create step counters as torch.zeros(1, dtype=torch.float, requires_grad=False), or let Adopt lazily initialize state itself","Detach step tensors: state['step'] = state['step'].detach() before stepping","Only mark exp_avg/exp_avg_sq (and params) as differentiable, not step"],"exampleFix":"# before\nstate['step'] = torch.zeros(1, requires_grad=True)\n# after\nstate['step'] = torch.zeros(1, requires_grad=False)","handlingStrategy":"validation","validationCode":"assert not any(p in opt.state and opt.state[p]['step'].requires_grad for p in group['params'])","typeGuard":"def step_counters_safe(opt) -> bool:\n    return all(not st['step'].requires_grad for st in opt.state.values() if 'step' in st)","tryCatchPattern":null,"preventionTips":["Never set requires_grad=True on step counters","Let Adopt initialize its own state lazily","In differentiable mode, only differentiate through moments and params"],"tags":["optimizer","adopt","differentiable","autograd","meta-learning"],"backgroundTag":"optimizer-differentiable-step-requires-grad","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}