huggingface/pytorch-image-models · error · RuntimeError

ADOPT does not support sparse gradients

Error message

ADOPT does not support sparse gradients

What it means

ADOPT's update kernels only operate on dense gradients; the per-parameter group scan raises as soon as a parameter's gradient is a sparse tensor, before any state initialization.

Source

Thrown at timm/optim/adopt.py:155

                    )

    def _init_group(
            self,
            group,
            params_with_grad,
            grads,
            exp_avgs,
            exp_avg_sqs,
            state_steps,
    ):
        has_complex = False
        for p in group["params"]:
            if p.grad is None:
                continue
            has_complex |= torch.is_complex(p)
            params_with_grad.append(p)
            if p.grad.is_sparse:
                raise RuntimeError("ADOPT does not support sparse gradients")
            grads.append(p.grad)

            state = self.state[p]
            # Lazy state initialization
            if len(state) == 0:
                # note(crcrpar): [special device hosting for step]
                # Deliberately host `step` on CPU if both capturable and fused are off.
                # This is because kernel launches are costly on CUDA and XLA.
                state["step"] = (
                    torch.zeros((), dtype=_get_scalar_dtype(), device=p.grad.device)
                    if group["capturable"]
                    else torch.tensor(0.0, dtype=_get_scalar_dtype())
                )
                # Exponential moving average of gradient values
                state["exp_avg"] = torch.zeros_like(p.grad, memory_format=torch.preserve_format)
                # Exponential moving average of squared gradient values
                state["exp_avg_sq"] = torch.zeros_like(p.grad, memory_format=torch.preserve_format)

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Remove sparse=True from nn.Embedding layers so gradients are dense
2. Use a sparse-capable optimizer (e.g. torch.optim.SparseAdam) for embedding parameters, keeping Adopt for the dense remainder
3. Wrap sparse params in a separate param group handled by a different optimizer

Example fix

# before
emb = nn.Embedding(vocab, dim, sparse=True)
opt = timm.optim.Adopt(model.parameters())
# after
emb = nn.Embedding(vocab, dim)
opt = timm.optim.Adopt(model.parameters())

bad = [n for n, p in model.named_parameters() if p.grad is not None and p.grad.is_sparse]
assert not bad, f'sparse grads on: {bad}'

def has_sparse_grads(params) -> bool:
    return any(p.grad is not None and p.grad.is_sparse for p in params)

try:
    optimizer.step()
except RuntimeError as e:
    if 'sparse gradients' in str(e):
        # fall back to SparseAdam for embedding params
        ...

When it happens

Trigger: Calling optimizer.step() on an Adopt instance when a parameter's .grad.is_sparse is True — most commonly an nn.Embedding(sparse=True) in the model.

Common situations: NLP or recommendation models with sparse embeddings trained with timm's Adopt; migrating a model that previously used SparseAdam.

Related errors

• AdamW does not support sparse gradients
• lr as a Tensor is not supported for capturable=False and for
• Tensor lr must be 1-element
• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/b4a0dca8e569867d. Report an issue: GitHub.