What causes "AdamW does not support sparse gradients" in geekcomputers/Python?

Replace AdamW with torch.optim.SparseAdam (optionally chained with AdamW for dense params)

geekcomputers/Python · error · RuntimeError

AdamW does not support sparse gradients

Error message

AdamW does not support sparse gradients

What it means

This RuntimeError is raised inside AdamW.step() when a parameter's gradient tensor is sparse (e.g., a torch.sparse tensor). AdamW's update math requires dense elementwise operations on gradient, exp_avg, and exp_avg_sq buffers, which are not defined for sparse layouts, so the implementation explicitly rejects them instead of failing obscurely later.

Source

Thrown at ML/src/python/neuralforge/optim/optimizers.py:31

        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")
        
        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, amsgrad=amsgrad)
        super().__init__(params, defaults)
    
    def step(self, closure=None):
        loss = None
        if closure is not None:
            loss = closure()
        
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                
                grad = p.grad.data
                if grad.is_sparse:
                    raise RuntimeError('AdamW does not support sparse gradients')
                
                amsgrad = group['amsgrad']
                state = self.state[p]
                
                if len(state) == 0:
                    state['step'] = 0
                    state['exp_avg'] = torch.zeros_like(p.data)
                    state['exp_avg_sq'] = torch.zeros_like(p.data)
                    if amsgrad:
                        state['max_exp_avg_sq'] = torch.zeros_like(p.data)
                
                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
                if amsgrad:
                    max_exp_avg_sq = state['max_exp_avg_sq']
                beta1, beta2 = group['betas']
                
                state['step'] += 1

View on GitHub (pinned to 40f4cd2652)

Solutions

Replace AdamW with torch.optim.SparseAdam (optionally chained with AdamW for dense params)
Set sparse=False on the offending nn.Embedding and re-create the optimizer
If you control the loss, avoid loss functions/backward paths that produce sparse gradients

Example fix

# before
emb = nn.Embedding(10000, 128, sparse=True)
opt = AdamW(model.parameters(), lr=1e-3)
opt.step()  # RuntimeError

# after
emb = nn.Embedding(10000, 128, sparse=True)
sparse_params = [emb.weight]
dense_params = [p for n, p in model.named_parameters() if n != 'emb.weight']
opt = SparseAdam(sparse_params, lr=1e-3)
opt2 = AdamW(dense_params, lr=1e-3)
loss.backward()
opt.step(); opt2.step()

Defensive patterns

Strategy: type-guard

Validate before calling

sparse_params = [p for p in model.parameters() if p.grad is not None and p.grad.is_sparse]
if sparse_params:
    opt = SparseAdam(sparse_params, lr=1e-3)
else:
    opt = AdamW(model.parameters(), lr=1e-3)

Type guard

def has_sparse_grads(model) -> bool:
    return any(p.grad is not None and p.grad.is_sparse for p in model.parameters())

Try / catch

try:
    opt.step()
except RuntimeError as e:
    if 'sparse gradients' in str(e):
        # rebuild optimizer split into SparseAdam + AdamW
        raise
    raise

Prevention

Use torch.optim.SparseAdam whenever any embedding has sparse=True
Document which parameters produce sparse gradients when designing the training loop
Wrap optimizer construction in a helper that inspects grads and routes sparse params correctly

When it happens

Trigger: Calling optimizer.step() after a backward pass where some parameter's .grad is sparse — typically embeddings with sparse=True (nn.Embedding(sparse=True)), sparse MSELoss, or manual assignment of sparse gradients. The check `if grad.is_sparse` fires per-parameter inside the step loop.

Common situations: Using nn.Embedding(sparse=True) with AdamW (common in NLP/recommender models to save memory on large vocabularies), then switching optimizer from SparseAdam to AdamW without changing the embedding, or building custom autograd Functions that return sparse grads.

Related errors

AI-assisted analysis of geekcomputers/Python@40f4cd2652 (2026-08-27). Data as JSON: /api/errors/86de28bd5a11c331. Report an issue: GitHub.