huggingface/pytorch-image-models · error · RuntimeError

weight_decay option is not compatible with sparse gradients

Error message

weight_decay option is not compatible with sparse gradients

What it means

In step(), MADGRAD with coupled (non-decoupled) weight decay adds weight_decay * p into the gradient, which cannot be done on a sparse tensor, so sparse gradients plus nonzero coupled weight decay raise a RuntimeError.

Source

Thrown at timm/optim/madgrad.py:135

                if len(state) == 0:
                    state['step'] = 0
                    state['grad_sum_sq'] = torch.zeros_like(p)
                    state['s'] = torch.zeros_like(p)
                    if momentum != 0:
                        state['x0'] = torch.clone(p).detach()

                state['step'] += 1
                grad_sum_sq = state['grad_sum_sq']
                s = state['s']
                lamb = lr * math.sqrt(state['step'])

                # Apply weight decay
                if weight_decay != 0:
                    if group['decoupled_decay']:
                        p.mul_(1.0 - group['lr'] * weight_decay)
                    else:
                        if grad.is_sparse:
                            raise RuntimeError("weight_decay option is not compatible with sparse gradients")
                        grad.add_(p, alpha=weight_decay)

                if grad.is_sparse:
                    grad = grad.coalesce()
                    grad_val = grad._values()

                    p_masked = p.sparse_mask(grad)
                    grad_sum_sq_masked = grad_sum_sq.sparse_mask(grad)
                    s_masked = s.sparse_mask(grad)

                    # Compute x_0 from other known quantities
                    rms_masked_vals = grad_sum_sq_masked._values().pow(1 / 3).add_(eps)
                    x0_masked_vals = p_masked._values().addcdiv(s_masked._values(), rms_masked_vals, value=1)

                    # Dense + sparse op
                    grad_sq = grad * grad
                    grad_sum_sq.add_(grad_sq, alpha=lamb)
                    grad_sum_sq_masked.add_(grad_sq, alpha=lamb)

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Set decoupled_decay=True for the sparse group (decoupled decay multiplies params directly and works with sparse grads)
2. Set weight_decay=0 for the sparse param group
3. Make gradients dense or use a sparse-capable optimizer

Example fix

# before
opt = MADGRAD([{'params': emb.parameters()}], lr=1e-3, weight_decay=1e-4)
# after
opt = MADGRAD([{'params': emb.parameters(), 'decoupled_decay': True}], lr=1e-3, weight_decay=1e-4)

assert decoupled_decay or weight_decay == 0 or not any(
    p.grad is not None and p.grad.is_sparse for p in sparse_params)

try:
    opt.step()
except RuntimeError as e:
    if 'weight_decay' in str(e) and 'sparse' in str(e):
        for g in opt.param_groups: g['decoupled_decay'] = True
    else:
        raise

When it happens

Trigger: A param group with weight_decay != 0 and decoupled_decay=False whose parameter receives a sparse gradient, when step() runs.

Common situations: Sparse embedding training where the config also enables L2 weight decay; default MADGRAD settings use coupled decay.

Related errors

• Weight decay {weight_decay} must be non-negative
• momentum != 0 is not compatible with sparse gradients
• AdamW does not support sparse gradients
• Invalid weight_decay value: {weight_decay}
• ADOPT does not support sparse gradients

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/c8439e443ab28b33. Report an issue: GitHub.