{"record":{"id":"d4aea74c602e70ca","repo":"labmlai/annotated_deep_learning_paper_implementations","slug":"genericadaptiveoptimizer-does-not-support-sparse-g","errorCode":null,"errorMessage":"GenericAdaptiveOptimizer does not support sparse gradients, please consider SparseAdam instead","messagePattern":"GenericAdaptiveOptimizer does not support sparse gradients, please consider SparseAdam instead","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"labml_nn/optimizers/__init__.py","lineNumber":150,"sourceCode":"        # calculates the loss, does `loss.backward` and return the loss, instead of calling\n        # it on your own you could pass it to `optimizer.step`. 🤷‍♂️\n        loss = None\n        if closure is not None:\n            with torch.enable_grad():\n                loss = closure()\n\n        # Iterate through the parameter groups\n        for group in self.param_groups:\n            # Iterate through the parameters in the parameter group\n            for param in group['params']:\n                # Skip if the parameter has no gradient\n                if param.grad is None:\n                    continue\n                # Get the gradient tensor\n                grad = param.grad.data\n                # We don't handle sparse gradients\n                if grad.is_sparse:\n                    raise RuntimeError('GenericAdaptiveOptimizer does not support sparse gradients,'\n                                       ' please consider SparseAdam instead')\n\n                # Get the state for the parameter\n                state = self.state[param]\n\n                # Initialize the state if state is uninitialized\n                if len(state) == 0:\n                    self.init_state(state, group, param)\n\n                # Take the optimization step on the parameter\n                self.step_param(state, group, grad, param)\n\n        # Return the loss, calculated from closure\n        return loss\n\n\nclass WeightDecay:\n    \"\"\"","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/33ab02281c2b928e6b32792909cc79cbdcfe1d6a/labml_nn/optimizers/__init__.py#L132-L168","documentation":"GenericAdaptiveOptimizer.step() iterates over gradients and explicitly refuses sparse gradient tensors (grad.is_sparse). The adaptive per-parameter state machinery assumes dense tensors; sparse embeddings would corrupt it. The error message points you to torch.optim.SparseAdam, which implements the sparse-safe Adam update.","triggerScenarios":"Calling optimizer.step() when any param.grad is sparse — classically nn.Embedding(..., sparse=True) producing sparse grads; or a custom autograd Function returning torch.sparse tensors; typically surfaces on the first step of training.","commonSituations":"Adding an embedding layer with sparse=True for memory efficiency with large vocabularies; switching from torch.optim.Adam (which errors differently or tolerates some cases via SparseAdam guidance) to a labml-nn Adam variant; GPT-NeoX runs with sparse embedding gradients enabled by default in some configs.","solutions":["Set sparse=False on the nn.Embedding (grads become dense; simplest correct fix)","For sparse embeddings, use torch.optim.SparseAdam on the sparse params (possibly a second optimizer instance for dense params)","Remove the custom sparse autograd path or call .to_dense() on gradients before stepping"],"exampleFix":"# before\nself.embedding = nn.Embedding(n_vocab, d_embed, sparse=True)\n\n# after\nself.embedding = nn.Embedding(n_vocab, d_embed, sparse=False)\n\n# or: keep sparse grads but step them with SparseAdam\nopt_sparse = torch.optim.SparseAdam(sparse_params, lr=1e-3)","handlingStrategy":"try-catch","validationCode":"for module in model.modules():\n    if isinstance(module, nn.Embedding) and module.sparse:\n        raise SystemExit('Sparse embedding found; set sparse=False or use SparseAdam')","typeGuard":"def params_all_dense(model: nn.Module) -> bool:\n    return not any(getattr(p, 'is_sparse', False) for p in model.parameters())","tryCatchPattern":"try:\n    opt.step()\nexcept RuntimeError as e:\n    if 'sparse gradients' in str(e):\n        raise SystemExit('Switch sparse embeddings to sparse=False, or step them with torch.optim.SparseAdam')\n    raise","preventionTips":["Default nn.Embedding to sparse=False unless memory forces otherwise","If using sparse=True, route those params to SparseAdam and the rest to the labml-nn optimizer","Add a model-audit step before training that scans for sparse embeddings"],"tags":["python","pytorch","optimizer","sparse-gradients","embedding"],"backgroundTag":"sparse-gradient-unsupported","analyzedSha":"33ab02281c2b928e6b32792909cc79cbdcfe1d6a","analyzedAt":"2026-08-25T10:30:27.743Z","schemaVersion":2},"datasetVersion":"2026-08-25T11:17:15.655Z"}