{"record":{"id":"3def774fadc7fc4e","repo":"huggingface/pytorch-image-models","slug":"adamw-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"AdamW does not support sparse gradients","messagePattern":"AdamW does not support sparse gradients","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/adamw.py","lineNumber":135,"sourceCode":"            with torch.enable_grad():\n                loss = closure()\n\n        for group in self.param_groups:\n            params_with_grad = []\n            grads = []\n            exp_avgs = []\n            exp_avg_sqs = []\n            max_exp_avg_sqs = []\n            state_steps = []\n            beta1, beta2 = group['betas']\n            amsgrad = group['amsgrad']\n\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                params_with_grad.append(p)\n                if p.grad.is_sparse:\n                    raise RuntimeError('AdamW does not support sparse gradients')\n                grads.append(p.grad)\n\n                state = self.state[p]\n\n                # State initialization\n                if len(state) == 0:\n                    state['step'] = _init_scalar(device=p.device if group['capturable'] else 'cpu')\n                    # Exponential moving average of gradient values\n                    state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n                    # Exponential moving average of squared gradient values\n                    state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n                    if amsgrad:\n                        # Maintains max of all exp. moving avg. of sq. grad. values\n                        state['max_exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n\n                exp_avgs.append(state['exp_avg'])\n                exp_avg_sqs.append(state['exp_avg_sq'])\n                if amsgrad:","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/adamw.py#L117-L153","documentation":"AdamW's update kernel only implements dense tensor math (element-wise ops, torch._foreach_*), so it cannot process torch sparse gradient tensors. The step() method raises immediately upon encountering a parameter whose .grad.is_sparse is True.","triggerScenarios":"Calling optimizer.step() when at least one parameter has a sparse gradient, typically from nn.Embedding with sparse=True (common in NLP/recommendation models) trained with timm.optim.AdamW.","commonSituations":"Using an nn.Embedding(sparse=True) layer (or torch.sparse gradients from backward) while using timm's AdamW; switching a model to sparse embeddings without changing the optimizer to one that supports sparse gradients (e.g. torch.optim.SparseAdam or standard torch.optim.AdamW).","solutions":["Remove sparse=True from your nn.Embedding so gradients are dense (fine for small vocabularies)","Switch to torch.optim.SparseAdam (or torch.optim.AdamW, which handles sparse embedding grads) for the sparse parameters","Keep sparse-parameter groups under a sparse-capable optimizer and the rest under AdamW via per-param groups"],"exampleFix":"# before\nemb = nn.Embedding(vocab, dim, sparse=True)\nopt = timm.optim.AdamW(model.parameters())\n# after\nemb = nn.Embedding(vocab, dim)  # dense grads\nopt = timm.optim.AdamW(model.parameters())","handlingStrategy":"validation","validationCode":"bad = [n for n, p in model.named_parameters() if p.grad is not None and p.grad.is_sparse]\nassert not bad, f'sparse grads on: {bad}'","typeGuard":"def has_sparse_grads(params) -> bool:\n    return any(p.grad is not None and p.grad.is_sparse for p in params)","tryCatchPattern":"try:\n    optimizer.step()\nexcept RuntimeError as e:\n    if 'sparse gradients' in str(e):\n        # switch embedding params to SparseAdam or make grads dense\n        ...","preventionTips":["Avoid nn.Embedding(sparse=True) unless the optimizer supports sparse grads","Split sparse/dense params into separate optimizers","Check p.grad.is_sparse in a debug pass after the first backward"],"tags":["optimizer","adamw","sparse-gradients","embedding"],"backgroundTag":"optimizer-sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}