huggingface/pytorch-image-models · error · RuntimeError

RAdam does not support sparse gradients

Error message

RAdam does not support sparse gradients

What it means

RuntimeError raised in RAdam.step() when a parameter gradient is a sparse tensor. RAdam (rectified Adam) only supports dense gradients; it also casts grads to float32, which sparse grads would complicate.

Source

Thrown at timm/optim/radam.py:51

    def __setstate__(self, state):
        super(RAdamLegacy, self).__setstate__(state)

    @torch.no_grad()
    def step(self, closure=None):
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:

            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad.float()
                if grad.is_sparse:
                    raise RuntimeError('RAdam does not support sparse gradients')

                p_fp32 = p.float()

                state = self.state[p]

                if len(state) == 0:
                    state['step'] = 0
                    state['exp_avg'] = torch.zeros_like(p_fp32)
                    state['exp_avg_sq'] = torch.zeros_like(p_fp32)
                else:
                    state['exp_avg'] = state['exp_avg'].type_as(p_fp32)
                    state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_fp32)

                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
                beta1, beta2 = group['betas']

                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
                exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Set sparse=False on embedding layers
2. Move sparse params to torch.optim.SparseAdam while keeping RAdam for dense params
3. Avoid assigning sparse tensors to .grad manually

Example fix

# before
emb = nn.Embedding(vocab, dim, sparse=True)
opt = RAdam(model.parameters())

# after
emb = nn.Embedding(vocab, dim, sparse=False)
opt = RAdam(model.parameters())

assert all(p.grad is None or not p.grad.is_sparse for p in model.parameters())

When it happens

Trigger: A model containing nn.Embedding(sparse=True) (or manual sparse grad assignment) trained with timm.optim.RAdam, then optimizer.step().

Common situations: NLP models with sparse embeddings switched to RAdam; mixed optimizer setups where embedding params were grouped with the rest of the model.

Related errors

• NAdamW does not support sparse gradients
• Sparse gradients are not supported.
• RMSprop does not support sparse gradients
• AdamW does not support sparse gradients
• ADOPT does not support sparse gradients

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/7a29dd65c6ac53b0. Report an issue: GitHub.