huggingface/pytorch-image-models · error · RuntimeError

Sparse gradients are not supported.

Error message

Sparse gradients are not supported.

What it means

RuntimeError raised in Nvnovograd.step() when a parameter gradient is a sparse torch tensor; this optimizer implements only dense updates.

Source

Thrown at timm/optim/nvnovograd.py:85

    def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
            and returns the loss.
        """
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad
                if grad.is_sparse:
                    raise RuntimeError('Sparse gradients are not supported.')
                amsgrad = group['amsgrad']

                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = 0
                    # Exponential moving average of gradient values
                    state['exp_avg'] = torch.zeros_like(p)
                    # Exponential moving average of squared gradient values
                    state['exp_avg_sq'] = torch.zeros([]).to(state['exp_avg'].device)
                    if amsgrad:
                        # Maintains max of all exp. moving avg. of sq. grad. values
                        state['max_exp_avg_sq'] = torch.zeros([]).to(state['exp_avg'].device)

                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
                if amsgrad:
                    max_exp_avg_sq = state['max_exp_avg_sq']

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Create embeddings with sparse=False
2. Use a sparse-capable optimizer (e.g. SparseAdam) for sparse params
3. Split sparse and dense params into separate optimizers

Example fix

# before
emb = nn.Embedding(vocab, dim, sparse=True)
opt = Nvnovograd(model.parameters())

# after
emb = nn.Embedding(vocab, dim, sparse=False)
opt = Nvnovograd(model.parameters())

assert all(p.grad is None or not p.grad.is_sparse for p in model.parameters())

When it happens

Trigger: Backward pass producing sparse grads (nn.Embedding(sparse=True)) followed by Nvnovograd.step().

Common situations: Training text/tabular models with sparse embeddings using timm's Nvnovograd, or porting a model from an optimizer that allowed sparsity.

Related errors

• NAdamW does not support sparse gradients
• RAdam does not support sparse gradients
• RMSprop does not support sparse gradients
• AdamW does not support sparse gradients
• ADOPT does not support sparse gradients

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/4a485074f78e2b9e. Report an issue: GitHub.