{"record":{"id":"ff16864d3b1d60ae","repo":"huggingface/pytorch-image-models","slug":"momentum-0-is-not-compatible-with-sparse-gradie","errorCode":null,"errorMessage":"momentum != 0 is not compatible with sparse gradients","messagePattern":"momentum != 0 is not compatible with sparse gradients","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/madgrad.py","lineNumber":114,"sourceCode":"        \"\"\"\n        loss = None\n        if closure is not None:\n            with torch.enable_grad():\n                loss = closure()\n\n        for group in self.param_groups:\n            eps = group['eps']\n            lr = group['lr'] + eps\n            weight_decay = group['weight_decay']\n            momentum = group['momentum']\n            ck = 1 - momentum\n\n            for p in group[\"params\"]:\n                if p.grad is None:\n                    continue\n                grad = p.grad\n                if momentum != 0.0 and grad.is_sparse:\n                    raise RuntimeError(\"momentum != 0 is not compatible with sparse gradients\")\n\n                state = self.state[p]\n                if len(state) == 0:\n                    state['step'] = 0\n                    state['grad_sum_sq'] = torch.zeros_like(p)\n                    state['s'] = torch.zeros_like(p)\n                    if momentum != 0:\n                        state['x0'] = torch.clone(p).detach()\n\n                state['step'] += 1\n                grad_sum_sq = state['grad_sum_sq']\n                s = state['s']\n                lamb = lr * math.sqrt(state['step'])\n\n                # Apply weight decay\n                if weight_decay != 0:\n                    if group['decoupled_decay']:\n                        p.mul_(1.0 - group['lr'] * weight_decay)","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/madgrad.py#L96-L132","documentation":"During step(), MADGRAD refuses to apply momentum to sparse gradient tensors because the momentum buffer update requires dense element-wise operations; combining nonzero momentum with sparse grads is unsupported.","triggerScenarios":"Calling optimizer.step() on parameters whose .grad is a torch sparse tensor (e.g. from an embedding backward with sparse=True) while momentum != 0.0 in the param group.","commonSituations":"Training models with sparse embedding gradients (NLP/recommendation) using MADGRAD with its default momentum=0.9.","solutions":["Set momentum=0 for the sparse param group (MADGRAD supports sparse grads only without momentum)","Produce dense gradients instead (remove sparse=True from the embedding or use torch.sparse.sum-style dense conversion)","Switch to an optimizer with sparse support, e.g. timm/PyTorch SparseAdam-style variants"],"exampleFix":"# before\nopt = MADGRAD([{'params': emb.parameters(), 'momentum': 0.9}], lr=1e-3)\n# after\nopt = MADGRAD([{'params': emb.parameters(), 'momentum': 0.0}], lr=1e-3)","handlingStrategy":"validation","validationCode":"sparse = any(p.grad is not None and p.grad.is_sparse for p in params)\nassert not (sparse and momentum != 0), 'sparse grads require momentum=0 in MADGRAD'","typeGuard":"def grads_are_dense(params) -> bool:\n    return all(p.grad is None or not p.grad.is_sparse for p in params)","tryCatchPattern":"try:\n    opt.step()\nexcept RuntimeError as e:\n    if 'sparse' in str(e):\n        for g in opt.param_groups: g['momentum'] = 0.0\n    else:\n        raise","preventionTips":["Use separate param groups for embeddings","Set momentum=0 for sparse param groups","Avoid sparse=True embeddings with MADGRAD momentum"],"tags":["optimizer","madgrad","sparse-gradients","momentum"],"backgroundTag":"sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}