huggingface/pytorch-image-models · error · RuntimeError

AdamW does not support sparse gradients

Error message

AdamW does not support sparse gradients

What it means

AdamW's update kernel only implements dense tensor math (element-wise ops, torch._foreach_*), so it cannot process torch sparse gradient tensors. The step() method raises immediately upon encountering a parameter whose .grad.is_sparse is True.

Source

Thrown at timm/optim/adamw.py:135

            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            params_with_grad = []
            grads = []
            exp_avgs = []
            exp_avg_sqs = []
            max_exp_avg_sqs = []
            state_steps = []
            beta1, beta2 = group['betas']
            amsgrad = group['amsgrad']

            for p in group['params']:
                if p.grad is None:
                    continue
                params_with_grad.append(p)
                if p.grad.is_sparse:
                    raise RuntimeError('AdamW does not support sparse gradients')
                grads.append(p.grad)

                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = _init_scalar(device=p.device if group['capturable'] else 'cpu')
                    # Exponential moving average of gradient values
                    state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    # Exponential moving average of squared gradient values
                    state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    if amsgrad:
                        # Maintains max of all exp. moving avg. of sq. grad. values
                        state['max_exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)

                exp_avgs.append(state['exp_avg'])
                exp_avg_sqs.append(state['exp_avg_sq'])
                if amsgrad:

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Remove sparse=True from your nn.Embedding so gradients are dense (fine for small vocabularies)
2. Switch to torch.optim.SparseAdam (or torch.optim.AdamW, which handles sparse embedding grads) for the sparse parameters
3. Keep sparse-parameter groups under a sparse-capable optimizer and the rest under AdamW via per-param groups

Example fix

# before
emb = nn.Embedding(vocab, dim, sparse=True)
opt = timm.optim.AdamW(model.parameters())
# after
emb = nn.Embedding(vocab, dim)  # dense grads
opt = timm.optim.AdamW(model.parameters())

bad = [n for n, p in model.named_parameters() if p.grad is not None and p.grad.is_sparse]
assert not bad, f'sparse grads on: {bad}'

def has_sparse_grads(params) -> bool:
    return any(p.grad is not None and p.grad.is_sparse for p in params)

try:
    optimizer.step()
except RuntimeError as e:
    if 'sparse gradients' in str(e):
        # switch embedding params to SparseAdam or make grads dense
        ...

When it happens

Trigger: Calling optimizer.step() when at least one parameter has a sparse gradient, typically from nn.Embedding with sparse=True (common in NLP/recommendation models) trained with timm.optim.AdamW.

Common situations: Using an nn.Embedding(sparse=True) layer (or torch.sparse gradients from backward) while using timm's AdamW; switching a model to sparse embeddings without changing the optimizer to one that supports sparse gradients (e.g. torch.optim.SparseAdam or standard torch.optim.AdamW).

Related errors

• ADOPT does not support sparse gradients
• Invalid beta parameter at index 0: {}
• Invalid beta parameter at index 1: {}
• API has changed, `state_steps` argument must contain a list
• momentum != 0 is not compatible with sparse gradients

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/3def774fadc7fc4e. Report an issue: GitHub.