{"record":{"id":"7a29dd65c6ac53b0","repo":"huggingface/pytorch-image-models","slug":"radam-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"RAdam does not support sparse gradients","messagePattern":"RAdam does not support sparse gradients","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/radam.py","lineNumber":51,"sourceCode":"\n    def __setstate__(self, state):\n        super(RAdamLegacy, self).__setstate__(state)\n\n    @torch.no_grad()\n    def step(self, closure=None):\n        loss = None\n        if closure is not None:\n            with torch.enable_grad():\n                loss = closure()\n\n        for group in self.param_groups:\n\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                grad = p.grad.float()\n                if grad.is_sparse:\n                    raise RuntimeError('RAdam does not support sparse gradients')\n\n                p_fp32 = p.float()\n\n                state = self.state[p]\n\n                if len(state) == 0:\n                    state['step'] = 0\n                    state['exp_avg'] = torch.zeros_like(p_fp32)\n                    state['exp_avg_sq'] = torch.zeros_like(p_fp32)\n                else:\n                    state['exp_avg'] = state['exp_avg'].type_as(p_fp32)\n                    state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_fp32)\n\n                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']\n                beta1, beta2 = group['betas']\n\n                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)\n                exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/radam.py#L33-L69","documentation":"RuntimeError raised in RAdam.step() when a parameter gradient is a sparse tensor. RAdam (rectified Adam) only supports dense gradients; it also casts grads to float32, which sparse grads would complicate.","triggerScenarios":"A model containing nn.Embedding(sparse=True) (or manual sparse grad assignment) trained with timm.optim.RAdam, then optimizer.step().","commonSituations":"NLP models with sparse embeddings switched to RAdam; mixed optimizer setups where embedding params were grouped with the rest of the model.","solutions":["Set sparse=False on embedding layers","Move sparse params to torch.optim.SparseAdam while keeping RAdam for dense params","Avoid assigning sparse tensors to .grad manually"],"exampleFix":"# before\nemb = nn.Embedding(vocab, dim, sparse=True)\nopt = RAdam(model.parameters())\n\n# after\nemb = nn.Embedding(vocab, dim, sparse=False)\nopt = RAdam(model.parameters())","handlingStrategy":"validation","validationCode":"assert all(p.grad is None or not p.grad.is_sparse for p in model.parameters())","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set sparse=False on embeddings before using RAdam","Segregate embedding params into SparseAdam groups"],"tags":["timm","radam","sparse-gradients","embedding"],"backgroundTag":"optimizer-sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}