huggingface/pytorch-image-models · error · RuntimeError

momentum != 0 is not compatible with sparse gradients

Error message

momentum != 0 is not compatible with sparse gradients

What it means

During step(), MADGRAD refuses to apply momentum to sparse gradient tensors because the momentum buffer update requires dense element-wise operations; combining nonzero momentum with sparse grads is unsupported.

Source

Thrown at timm/optim/madgrad.py:114

        """
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            eps = group['eps']
            lr = group['lr'] + eps
            weight_decay = group['weight_decay']
            momentum = group['momentum']
            ck = 1 - momentum

            for p in group["params"]:
                if p.grad is None:
                    continue
                grad = p.grad
                if momentum != 0.0 and grad.is_sparse:
                    raise RuntimeError("momentum != 0 is not compatible with sparse gradients")

                state = self.state[p]
                if len(state) == 0:
                    state['step'] = 0
                    state['grad_sum_sq'] = torch.zeros_like(p)
                    state['s'] = torch.zeros_like(p)
                    if momentum != 0:
                        state['x0'] = torch.clone(p).detach()

                state['step'] += 1
                grad_sum_sq = state['grad_sum_sq']
                s = state['s']
                lamb = lr * math.sqrt(state['step'])

                # Apply weight decay
                if weight_decay != 0:
                    if group['decoupled_decay']:
                        p.mul_(1.0 - group['lr'] * weight_decay)

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Set momentum=0 for the sparse param group (MADGRAD supports sparse grads only without momentum)
2. Produce dense gradients instead (remove sparse=True from the embedding or use torch.sparse.sum-style dense conversion)
3. Switch to an optimizer with sparse support, e.g. timm/PyTorch SparseAdam-style variants

Example fix

# before
opt = MADGRAD([{'params': emb.parameters(), 'momentum': 0.9}], lr=1e-3)
# after
opt = MADGRAD([{'params': emb.parameters(), 'momentum': 0.0}], lr=1e-3)

sparse = any(p.grad is not None and p.grad.is_sparse for p in params)
assert not (sparse and momentum != 0), 'sparse grads require momentum=0 in MADGRAD'

def grads_are_dense(params) -> bool:
    return all(p.grad is None or not p.grad.is_sparse for p in params)

try:
    opt.step()
except RuntimeError as e:
    if 'sparse' in str(e):
        for g in opt.param_groups: g['momentum'] = 0.0
    else:
        raise

When it happens

Trigger: Calling optimizer.step() on parameters whose .grad is a torch sparse tensor (e.g. from an embedding backward with sparse=True) while momentum != 0.0 in the param group.

Common situations: Training models with sparse embedding gradients (NLP/recommendation) using MADGRAD with its default momentum=0.9.

Related errors

• Momentum {momentum} must be in the range [0,1]
• weight_decay option is not compatible with sparse gradients
• AdamW does not support sparse gradients
• ADOPT does not support sparse gradients
• Learning rate {lr} must be positive

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/ff16864d3b1d60ae. Report an issue: GitHub.