{"record":{"id":"d534837ad8727758","repo":"huggingface/pytorch-image-models","slug":"adam-does-not-support-sparse-gradients-please-con","errorCode":null,"errorMessage":"Adam does not support sparse gradients, please consider SparseAdam instead","messagePattern":"Adam does not support sparse gradients, please consider SparseAdam instead","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/mars.py","lineNumber":160,"sourceCode":"    def step(self, closure=None):\n        \"\"\"Performs a single optimization step.\n\n        Arguments:\n            closure (callable, optional): A closure that reevaluates the model\n                and returns the loss.\n        \"\"\"\n        loss = None\n        if closure is not None:\n            with torch.enable_grad():\n                loss = closure()\n\n        for group in self.param_groups:\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                grad = p.grad\n                if grad.is_sparse:\n                    raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead')\n\n                state = self.state[p]\n                # State initialization\n                if len(state) <= 1:\n                    state['step'] = 0\n                    # Exponential moving average of gradient values\n                    state['exp_avg'] = torch.zeros_like(p)\n                    # Last Gradient\n                    state['last_grad'] = torch.zeros_like(p)\n                    # Exponential moving average of squared gradient values\n                    state['exp_avg_sq'] = torch.zeros_like(p)\n\n                state['step'] += 1\n                step = state['step']\n                exp_avg = state['exp_avg']\n                exp_avg_sq = state['exp_avg_sq']\n                last_grad = state['last_grad']\n                lr = group['lr']","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/mars.py#L142-L178","documentation":"Mars (AdamW/Lion-style implementation) does not implement sparse gradient handling; encountering a parameter whose .grad is a sparse tensor during step() raises immediately, mirroring PyTorch Adam's behavior which suggests SparseAdam.","triggerScenarios":"optimizer.step() when some parameter's gradient is a torch sparse tensor — typically nn.Embedding(sparse=True) or nn.Linear on one-hot inputs producing sparse grads.","commonSituations":"NLP/recommendation training with sparse embeddings using the Mars optimizer; switching an existing sparse-embedding pipeline to Mars.","solutions":["Remove sparse=True from the embedding so gradients are dense","Switch that param group to torch.optim.SparseAdam (or another sparse-capable optimizer)","Keep embeddings in a separate param group handled by a different optimizer"],"exampleFix":"# before\nemb = nn.Embedding(num, dim, sparse=True)\nopt = Mars(model.parameters())\n# after\nemb = nn.Embedding(num, dim)  # dense grads\nopt = Mars(model.parameters())","handlingStrategy":"type-guard","validationCode":"assert all(p.grad is None or not p.grad.is_sparse for p in params), 'Mars cannot step on sparse gradients'","typeGuard":"def has_sparse_grads(params) -> bool:\n    return any(p.grad is not None and p.grad.is_sparse for p in params)","tryCatchPattern":"try:\n    opt.step()\nexcept RuntimeError as e:\n    if 'SparseAdam' in str(e):\n        # move embeddings to SparseAdam and re-run\n        raise NotImplementedError('split sparse params into SparseAdam group')\n    raise","preventionTips":["Avoid nn.Embedding(..., sparse=True) when using Mars","Split sparse params into a separate optimizer"],"tags":["optimizer","mars","sparse-gradients"],"backgroundTag":"sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}