{"record":{"id":"39e3a8a9aab5781e","repo":"huggingface/pytorch-image-models","slug":"nadamw-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"NAdamW does not support sparse gradients","messagePattern":"NAdamW does not support sparse gradients","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/nadamw.py","lineNumber":125,"sourceCode":"        loss = None\n        if closure is not None:\n            with torch.enable_grad():\n                loss = closure()\n\n        for group in self.param_groups:\n            params_with_grad = []\n            grads = []\n            exp_avgs = []\n            exp_avg_sqs = []\n            state_steps = []\n            beta1, beta2 = group['betas']\n\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                params_with_grad.append(p)\n                if p.grad.is_sparse:\n                    raise RuntimeError('NAdamW does not support sparse gradients')\n                grads.append(p.grad)\n\n                state = self.state[p]\n\n                # State initialization\n                if len(state) == 0:\n                    state['step'] = _init_scalar(device=p.device if group['capturable'] else 'cpu')\n                    # Exponential moving average of gradient values\n                    state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n                    # Exponential moving average of squared gradient values\n                    state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n\n                exp_avgs.append(state['exp_avg'])\n                exp_avg_sqs.append(state['exp_avg_sq'])\n                state_steps.append(state['step'])\n\n            nadamw(\n                params_with_grad,","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/nadamw.py#L107-L143","documentation":"RuntimeError raised in NAdamW.step() when a parameter's gradient is a torch sparse tensor. The NAdamW update kernels only handle dense gradients.","triggerScenarios":"Calling optimizer.step() after a backward pass where at least one param grad is sparse, typically from nn.Embedding(sparse=True) used with NAdamW.","commonSituations":"NLP/recommendation models with sparse=True embeddings trained with NAdamW; switching an existing training script from SparseAdam/optimizers that tolerate sparsity to timm's NAdamW.","solutions":["Set sparse=False on nn.Embedding layers (memory cost rises but gradients become dense)","Use torch.optim.SparseAdam or an optimizer that supports sparse grads for embedding params","Keep sparse-embedding params in a separate param group handled by a supporting optimizer"],"exampleFix":"# before\nemb = nn.Embedding(num, dim, sparse=True)\nopt = NAdamW(model.parameters(), lr=1e-3)\n\n# after\nemb = nn.Embedding(num, dim, sparse=False)\nopt = NAdamW(model.parameters(), lr=1e-3)","handlingStrategy":"validation","validationCode":"sparse_params = [n for n, p in model.named_parameters() if p.grad is not None and p.grad.is_sparse]\nassert not sparse_params, f'sparse grads on: {sparse_params}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Avoid sparse=True embeddings with NAdamW","Audit embedding modules when switching optimizers"],"tags":["timm","nadamw","sparse-gradients","embedding"],"backgroundTag":"optimizer-sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}