{"record":{"id":"86de28bd5a11c331","repo":"geekcomputers/Python","slug":"adamw-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"AdamW does not support sparse gradients","messagePattern":"AdamW does not support sparse gradients","errorType":"error_code","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"ML/src/python/neuralforge/optim/optimizers.py","lineNumber":31,"sourceCode":"        if not 0.0 <= betas[1] < 1.0:\n            raise ValueError(f\"Invalid beta parameter at index 1: {betas[1]}\")\n        \n        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, amsgrad=amsgrad)\n        super().__init__(params, defaults)\n    \n    def step(self, closure=None):\n        loss = None\n        if closure is not None:\n            loss = closure()\n        \n        for group in self.param_groups:\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                \n                grad = p.grad.data\n                if grad.is_sparse:\n                    raise RuntimeError('AdamW does not support sparse gradients')\n                \n                amsgrad = group['amsgrad']\n                state = self.state[p]\n                \n                if len(state) == 0:\n                    state['step'] = 0\n                    state['exp_avg'] = torch.zeros_like(p.data)\n                    state['exp_avg_sq'] = torch.zeros_like(p.data)\n                    if amsgrad:\n                        state['max_exp_avg_sq'] = torch.zeros_like(p.data)\n                \n                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']\n                if amsgrad:\n                    max_exp_avg_sq = state['max_exp_avg_sq']\n                beta1, beta2 = group['betas']\n                \n                state['step'] += 1\n                ","sourceCodeStart":13,"sourceCodeEnd":49,"githubUrl":"https://github.com/geekcomputers/Python/blob/40f4cd2652d75ef8e49d76e5c4d431d458712719/ML/src/python/neuralforge/optim/optimizers.py#L13-L49","documentation":"This RuntimeError is raised inside AdamW.step() when a parameter's gradient tensor is sparse (e.g., a torch.sparse tensor). AdamW's update math requires dense elementwise operations on gradient, exp_avg, and exp_avg_sq buffers, which are not defined for sparse layouts, so the implementation explicitly rejects them instead of failing obscurely later.","triggerScenarios":"Calling optimizer.step() after a backward pass where some parameter's .grad is sparse — typically embeddings with sparse=True (nn.Embedding(sparse=True)), sparse MSELoss, or manual assignment of sparse gradients. The check `if grad.is_sparse` fires per-parameter inside the step loop.","commonSituations":"Using nn.Embedding(sparse=True) with AdamW (common in NLP/recommender models to save memory on large vocabularies), then switching optimizer from SparseAdam to AdamW without changing the embedding, or building custom autograd Functions that return sparse grads.","solutions":["Replace AdamW with torch.optim.SparseAdam (optionally chained with AdamW for dense params)","Set sparse=False on the offending nn.Embedding and re-create the optimizer","If you control the loss, avoid loss functions/backward paths that produce sparse gradients"],"exampleFix":"# before\nemb = nn.Embedding(10000, 128, sparse=True)\nopt = AdamW(model.parameters(), lr=1e-3)\nopt.step()  # RuntimeError\n\n# after\nemb = nn.Embedding(10000, 128, sparse=True)\nsparse_params = [emb.weight]\ndense_params = [p for n, p in model.named_parameters() if n != 'emb.weight']\nopt = SparseAdam(sparse_params, lr=1e-3)\nopt2 = AdamW(dense_params, lr=1e-3)\nloss.backward()\nopt.step(); opt2.step()","handlingStrategy":"type-guard","validationCode":"sparse_params = [p for p in model.parameters() if p.grad is not None and p.grad.is_sparse]\nif sparse_params:\n    opt = SparseAdam(sparse_params, lr=1e-3)\nelse:\n    opt = AdamW(model.parameters(), lr=1e-3)","typeGuard":"def has_sparse_grads(model) -> bool:\n    return any(p.grad is not None and p.grad.is_sparse for p in model.parameters())","tryCatchPattern":"try:\n    opt.step()\nexcept RuntimeError as e:\n    if 'sparse gradients' in str(e):\n        # rebuild optimizer split into SparseAdam + AdamW\n        raise\n    raise","preventionTips":["Use torch.optim.SparseAdam whenever any embedding has sparse=True","Document which parameters produce sparse gradients when designing the training loop","Wrap optimizer construction in a helper that inspects grads and routes sparse params correctly"],"tags":["python","pytorch","optimizer","adamw","sparse-gradients","embedding"],"backgroundTag":"optimizer-sparse-gradient-unsupported","analyzedSha":"40f4cd2652d75ef8e49d76e5c4d431d458712719","analyzedAt":"2026-08-27T11:12:20.313Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}