{"record":{"id":"b4a0dca8e569867d","repo":"huggingface/pytorch-image-models","slug":"adopt-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"ADOPT does not support sparse gradients","messagePattern":"ADOPT does not support sparse gradients","errorType":"validation","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/adopt.py","lineNumber":155,"sourceCode":"                    )\n\n    def _init_group(\n            self,\n            group,\n            params_with_grad,\n            grads,\n            exp_avgs,\n            exp_avg_sqs,\n            state_steps,\n    ):\n        has_complex = False\n        for p in group[\"params\"]:\n            if p.grad is None:\n                continue\n            has_complex |= torch.is_complex(p)\n            params_with_grad.append(p)\n            if p.grad.is_sparse:\n                raise RuntimeError(\"ADOPT does not support sparse gradients\")\n            grads.append(p.grad)\n\n            state = self.state[p]\n            # Lazy state initialization\n            if len(state) == 0:\n                # note(crcrpar): [special device hosting for step]\n                # Deliberately host `step` on CPU if both capturable and fused are off.\n                # This is because kernel launches are costly on CUDA and XLA.\n                state[\"step\"] = (\n                    torch.zeros((), dtype=_get_scalar_dtype(), device=p.grad.device)\n                    if group[\"capturable\"]\n                    else torch.tensor(0.0, dtype=_get_scalar_dtype())\n                )\n                # Exponential moving average of gradient values\n                state[\"exp_avg\"] = torch.zeros_like(p.grad, memory_format=torch.preserve_format)\n                # Exponential moving average of squared gradient values\n                state[\"exp_avg_sq\"] = torch.zeros_like(p.grad, memory_format=torch.preserve_format)\n","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/adopt.py#L137-L173","documentation":"ADOPT's update kernels only operate on dense gradients; the per-parameter group scan raises as soon as a parameter's gradient is a sparse tensor, before any state initialization.","triggerScenarios":"Calling optimizer.step() on an Adopt instance when a parameter's .grad.is_sparse is True — most commonly an nn.Embedding(sparse=True) in the model.","commonSituations":"NLP or recommendation models with sparse embeddings trained with timm's Adopt; migrating a model that previously used SparseAdam.","solutions":["Remove sparse=True from nn.Embedding layers so gradients are dense","Use a sparse-capable optimizer (e.g. torch.optim.SparseAdam) for embedding parameters, keeping Adopt for the dense remainder","Wrap sparse params in a separate param group handled by a different optimizer"],"exampleFix":"# before\nemb = nn.Embedding(vocab, dim, sparse=True)\nopt = timm.optim.Adopt(model.parameters())\n# after\nemb = nn.Embedding(vocab, dim)\nopt = timm.optim.Adopt(model.parameters())","handlingStrategy":"validation","validationCode":"bad = [n for n, p in model.named_parameters() if p.grad is not None and p.grad.is_sparse]\nassert not bad, f'sparse grads on: {bad}'","typeGuard":"def has_sparse_grads(params) -> bool:\n    return any(p.grad is not None and p.grad.is_sparse for p in params)","tryCatchPattern":"try:\n    optimizer.step()\nexcept RuntimeError as e:\n    if 'sparse gradients' in str(e):\n        # fall back to SparseAdam for embedding params\n        ...","preventionTips":["Don't set sparse=True on embeddings when using Adopt","Audit model for sparse grad sources before choosing the optimizer"],"tags":["optimizer","adopt","sparse-gradients","embedding"],"backgroundTag":"optimizer-sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}