labmlai/annotated_deep_learning_paper_implementations · error · RuntimeError

GenericAdaptiveOptimizer does not support sparse gradients,

Error message

GenericAdaptiveOptimizer does not support sparse gradients, please consider SparseAdam instead

What it means

GenericAdaptiveOptimizer.step() iterates over gradients and explicitly refuses sparse gradient tensors (grad.is_sparse). The adaptive per-parameter state machinery assumes dense tensors; sparse embeddings would corrupt it. The error message points you to torch.optim.SparseAdam, which implements the sparse-safe Adam update.

Source

Thrown at labml_nn/optimizers/__init__.py:150

        # calculates the loss, does `loss.backward` and return the loss, instead of calling
        # it on your own you could pass it to `optimizer.step`. 🤷‍♂️
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        # Iterate through the parameter groups
        for group in self.param_groups:
            # Iterate through the parameters in the parameter group
            for param in group['params']:
                # Skip if the parameter has no gradient
                if param.grad is None:
                    continue
                # Get the gradient tensor
                grad = param.grad.data
                # We don't handle sparse gradients
                if grad.is_sparse:
                    raise RuntimeError('GenericAdaptiveOptimizer does not support sparse gradients,'
                                       ' please consider SparseAdam instead')

                # Get the state for the parameter
                state = self.state[param]

                # Initialize the state if state is uninitialized
                if len(state) == 0:
                    self.init_state(state, group, param)

                # Take the optimization step on the parameter
                self.step_param(state, group, grad, param)

        # Return the loss, calculated from closure
        return loss


class WeightDecay:
    """

View on GitHub (pinned to 33ab02281c)

Solutions

Set sparse=False on the nn.Embedding (grads become dense; simplest correct fix)
For sparse embeddings, use torch.optim.SparseAdam on the sparse params (possibly a second optimizer instance for dense params)
Remove the custom sparse autograd path or call .to_dense() on gradients before stepping

Example fix

# before
self.embedding = nn.Embedding(n_vocab, d_embed, sparse=True)

# after
self.embedding = nn.Embedding(n_vocab, d_embed, sparse=False)

# or: keep sparse grads but step them with SparseAdam
opt_sparse = torch.optim.SparseAdam(sparse_params, lr=1e-3)

Defensive patterns

Strategy: try-catch

Validate before calling

for module in model.modules():
    if isinstance(module, nn.Embedding) and module.sparse:
        raise SystemExit('Sparse embedding found; set sparse=False or use SparseAdam')

Type guard

def params_all_dense(model: nn.Module) -> bool:
    return not any(getattr(p, 'is_sparse', False) for p in model.parameters())

Try / catch

try:
    opt.step()
except RuntimeError as e:
    if 'sparse gradients' in str(e):
        raise SystemExit('Switch sparse embeddings to sparse=False, or step them with torch.optim.SparseAdam')
    raise

Prevention

Default nn.Embedding to sparse=False unless memory forces otherwise
If using sparse=True, route those params to SparseAdam and the rest to the labml-nn optimizer
Add a model-audit step before training that scans for sparse embeddings

When it happens

Trigger: Calling optimizer.step() when any param.grad is sparse — classically nn.Embedding(..., sparse=True) producing sparse grads; or a custom autograd Function returning torch.sparse tensors; typically surfaces on the first step of training.

Common situations: Adding an embedding layer with sparse=True for memory efficiency with large vocabularies; switching from torch.optim.Adam (which errors differently or tolerates some cases via SparseAdam guidance) to a labml-nn Adam variant; GPT-NeoX runs with sparse embedding gradients enabled by default in some configs.

Related errors

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/d4aea74c602e70ca. Report an issue: GitHub.