lllyasviel/ControlNet · error · RuntimeError

AdamW does not support sparse gradients

Error message

AdamW does not support sparse gradients

What it means

Raised in the step() method of the custom AdamW when a parameter's gradient is a sparse torch tensor (p.grad.is_sparse). Like the stock torch.optim.AdamW, this implementation uses dense elementwise ops and cannot update parameters whose gradients are stored in sparse format.

Source

Thrown at ldm/util.py:149

            params_with_grad = []
            grads = []
            exp_avgs = []
            exp_avg_sqs = []
            ema_params_with_grad = []
            state_sums = []
            max_exp_avg_sqs = []
            state_steps = []
            amsgrad = group['amsgrad']
            beta1, beta2 = group['betas']
            ema_decay = group['ema_decay']
            ema_power = group['ema_power']

            for p in group['params']:
                if p.grad is None:
                    continue
                params_with_grad.append(p)
                if p.grad.is_sparse:
                    raise RuntimeError('AdamW does not support sparse gradients')
                grads.append(p.grad)

                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = 0
                    # Exponential moving average of gradient values
                    state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    # Exponential moving average of squared gradient values
                    state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    if amsgrad:
                        # Maintains max of all exp. moving avg. of sq. grad. values
                        state['max_exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)
                    # Exponential moving average of parameter values
                    state['param_exp_avg'] = p.detach().float().clone()

                exp_avgs.append(state['exp_avg'])

View on GitHub (pinned to ed85cd1e25)

Solutions

1. Set sparse=False on the embedding (or remove the flag) so gradients are materialized densely
2. Switch that parameter group to torch.optim.SparseAdam (note: no EMA support) or use two optimizers, routing sparse-grad params to SparseAdam
3. If the parameter is frozen in practice, exclude it from the optimizer's param groups

Example fix

# before
self.tok_emb = nn.Embedding(vocab, dim, sparse=True)
# after
self.tok_emb = nn.Embedding(vocab, dim)  # dense grads work with this AdamW

def params_all_dense(params):
    return all(p.grad is None or not p.grad.is_sparse for p in params)

try:
    optimizer.step()
except RuntimeError as e:
    if 'sparse gradients' in str(e):
        # move sparse-grad params (embeddings) to SparseAdam or set sparse=False
        raise
    raise

When it happens

Trigger: Calling optimizer.step() (invoked by the trainer's after_train_iter) when a model parameter's .grad is a sparse tensor — typical with embeddings trained with sparse=True (e.g. nn.Embedding(..., sparse=True)) or when backward is called on torch.sparse intermediate results.

Common situations: Adding a text/token embedding with sparse=True to a latent-diffusion model whose training loop uses this custom AdamW, or porting NLP components into the diffusion stack; PyTorch optimizers like SparseAdam exist precisely for this case.

Related errors

• Invalid learning rate: {}
• Invalid epsilon value: {}
• Invalid beta parameter at index 0: {}
• Invalid beta parameter at index 1: {}
• Invalid weight_decay value: {}

AI-assisted analysis of lllyasviel/ControlNet@ed85cd1e25 (2026-08-27). Data as JSON: /api/errors/d54e71fc96bff30e. Report an issue: GitHub.