huggingface/pytorch-image-models · error · RuntimeError

Adam does not support sparse gradients, please consider Spar

Error message

Adam does not support sparse gradients, please consider SparseAdam instead

What it means

Mars (AdamW/Lion-style implementation) does not implement sparse gradient handling; encountering a parameter whose .grad is a sparse tensor during step() raises immediately, mirroring PyTorch Adam's behavior which suggests SparseAdam.

Source

Thrown at timm/optim/mars.py:160

    def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad
                if grad.is_sparse:
                    raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead')

                state = self.state[p]
                # State initialization
                if len(state) <= 1:
                    state['step'] = 0
                    # Exponential moving average of gradient values
                    state['exp_avg'] = torch.zeros_like(p)
                    # Last Gradient
                    state['last_grad'] = torch.zeros_like(p)
                    # Exponential moving average of squared gradient values
                    state['exp_avg_sq'] = torch.zeros_like(p)

                state['step'] += 1
                step = state['step']
                exp_avg = state['exp_avg']
                exp_avg_sq = state['exp_avg_sq']
                last_grad = state['last_grad']
                lr = group['lr']

View on GitHub (pinned to 9a5261e31b)

Solutions

Remove sparse=True from the embedding so gradients are dense
Switch that param group to torch.optim.SparseAdam (or another sparse-capable optimizer)
Keep embeddings in a separate param group handled by a different optimizer

Example fix

# before
emb = nn.Embedding(num, dim, sparse=True)
opt = Mars(model.parameters())
# after
emb = nn.Embedding(num, dim)  # dense grads
opt = Mars(model.parameters())

Defensive patterns

Strategy: type-guard

Validate before calling

assert all(p.grad is None or not p.grad.is_sparse for p in params), 'Mars cannot step on sparse gradients'

Type guard

def has_sparse_grads(params) -> bool:
    return any(p.grad is not None and p.grad.is_sparse for p in params)

Try / catch

try:
    opt.step()
except RuntimeError as e:
    if 'SparseAdam' in str(e):
        # move embeddings to SparseAdam and re-run
        raise NotImplementedError('split sparse params into SparseAdam group')
    raise

Prevention

Avoid nn.Embedding(..., sparse=True) when using Mars
Split sparse params into a separate optimizer

When it happens

Trigger: optimizer.step() when some parameter's gradient is a torch sparse tensor — typically nn.Embedding(sparse=True) or nn.Linear on one-hot inputs producing sparse grads.

Common situations: NLP/recommendation training with sparse embeddings using the Mars optimizer; switching an existing sparse-embedding pipeline to Mars.

Related errors

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/d534837ad8727758. Report an issue: GitHub.