huggingface/pytorch-image-models · error · RuntimeError

RMSprop does not support sparse gradients

Error message

RMSprop does not support sparse gradients

What it means

RuntimeError raised in RMSpropTF.step() when any parameter gradient is a sparse tensor. This TF-style RMSprop implementation performs dense-only in-place tensor math on square_avg/acc_grad buffers.

Source

Thrown at timm/optim/rmsprop_tf.py:118

    def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad
                if grad.is_sparse:
                    raise RuntimeError('RMSprop does not support sparse gradients')
                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = _init_scalar(device='cpu')
                    state['square_avg'] = torch.ones_like(p)  # PyTorch inits to zero
                    if group['momentum'] > 0:
                        state['momentum_buffer'] = torch.zeros_like(p)
                    if group['centered']:
                        state['grad_avg'] = torch.zeros_like(p)

                square_avg = state['square_avg']
                one_minus_alpha = 1. - group['alpha']

                state['step'].add_(1)

                if group['weight_decay'] != 0:
                    if group['decoupled_decay']:

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Set sparse=False on embedding layers
2. Use torch.optim.SparseAdam or torch.optim.SGD (which support sparse grads) for embedding params
3. Maintain separate optimizers for sparse vs dense parameter groups

Example fix

# before
emb = nn.Embedding(num_items, 64, sparse=True)
opt = RMSpropTF(model.parameters())

# after
emb = nn.Embedding(num_items, 64, sparse=False)
opt = RMSpropTF(model.parameters())

assert all(p.grad is None or not p.grad.is_sparse for p in model.parameters())

When it happens

Trigger: Training a model with sparse gradients (e.g. nn.Embedding(sparse=True)) using timm.optim.RMSpropTF and calling step().

Common situations: Recommendation/search-ranking models with large embedding tables migrated to timm's RMSpropTF; configs reused from SGD-with-sparse setups.

Related errors

• NAdamW does not support sparse gradients
• Sparse gradients are not supported.
• RAdam does not support sparse gradients
• AdamW does not support sparse gradients
• ADOPT does not support sparse gradients

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/29fb10b2564e69f9. Report an issue: GitHub.