{"record":{"id":"d54e71fc96bff30e","repo":"lllyasviel/ControlNet","slug":"adamw-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"AdamW does not support sparse gradients","messagePattern":"AdamW does not support sparse gradients","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"ldm/util.py","lineNumber":149,"sourceCode":"            params_with_grad = []\n            grads = []\n            exp_avgs = []\n            exp_avg_sqs = []\n            ema_params_with_grad = []\n            state_sums = []\n            max_exp_avg_sqs = []\n            state_steps = []\n            amsgrad = group['amsgrad']\n            beta1, beta2 = group['betas']\n            ema_decay = group['ema_decay']\n            ema_power = group['ema_power']\n\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                params_with_grad.append(p)\n                if p.grad.is_sparse:\n                    raise RuntimeError('AdamW does not support sparse gradients')\n                grads.append(p.grad)\n\n                state = self.state[p]\n\n                # State initialization\n                if len(state) == 0:\n                    state['step'] = 0\n                    # Exponential moving average of gradient values\n                    state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n                    # Exponential moving average of squared gradient values\n                    state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n                    if amsgrad:\n                        # Maintains max of all exp. moving avg. of sq. grad. values\n                        state['max_exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format)\n                    # Exponential moving average of parameter values\n                    state['param_exp_avg'] = p.detach().float().clone()\n\n                exp_avgs.append(state['exp_avg'])","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/lllyasviel/ControlNet/blob/ed85cd1e25a5ed592f7d8178495b4483de0331bf/ldm/util.py#L131-L167","documentation":"Raised in the step() method of the custom AdamW when a parameter's gradient is a sparse torch tensor (p.grad.is_sparse). Like the stock torch.optim.AdamW, this implementation uses dense elementwise ops and cannot update parameters whose gradients are stored in sparse format.","triggerScenarios":"Calling optimizer.step() (invoked by the trainer's after_train_iter) when a model parameter's .grad is a sparse tensor — typical with embeddings trained with sparse=True (e.g. nn.Embedding(..., sparse=True)) or when backward is called on torch.sparse intermediate results.","commonSituations":"Adding a text/token embedding with sparse=True to a latent-diffusion model whose training loop uses this custom AdamW, or porting NLP components into the diffusion stack; PyTorch optimizers like SparseAdam exist precisely for this case.","solutions":["Set sparse=False on the embedding (or remove the flag) so gradients are materialized densely","Switch that parameter group to torch.optim.SparseAdam (note: no EMA support) or use two optimizers, routing sparse-grad params to SparseAdam","If the parameter is frozen in practice, exclude it from the optimizer's param groups"],"exampleFix":"# before\nself.tok_emb = nn.Embedding(vocab, dim, sparse=True)\n# after\nself.tok_emb = nn.Embedding(vocab, dim)  # dense grads work with this AdamW","handlingStrategy":"validation","validationCode":"def params_all_dense(params):\n    return all(p.grad is None or not p.grad.is_sparse for p in params)","typeGuard":null,"tryCatchPattern":"try:\n    optimizer.step()\nexcept RuntimeError as e:\n    if 'sparse gradients' in str(e):\n        # move sparse-grad params (embeddings) to SparseAdam or set sparse=False\n        raise\n    raise","preventionTips":["Avoid nn.Embedding(..., sparse=True) in models trained with this AdamW","Check p.grad.is_sparse in a sanity pass before the first step","If sparse embeddings are required, split them into a separate SparseAdam optimizer"],"tags":["optimizer","adamw","sparse-gradients","pytorch","embeddings","diffusion-training"],"backgroundTag":"sparse-gradient-unsupported-by-optimizer","analyzedSha":"ed85cd1e25a5ed592f7d8178495b4483de0331bf","analyzedAt":"2026-08-27T12:58:54.167Z","schemaVersion":2},"datasetVersion":"2026-08-27T13:17:12.746Z"}