{"record":{"id":"4a485074f78e2b9e","repo":"huggingface/pytorch-image-models","slug":"sparse-gradients-are-not-supported","errorCode":null,"errorMessage":"Sparse gradients are not supported.","messagePattern":"Sparse gradients are not supported\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/nvnovograd.py","lineNumber":85,"sourceCode":"    def step(self, closure=None):\n        \"\"\"Performs a single optimization step.\n\n        Arguments:\n            closure (callable, optional): A closure that reevaluates the model\n            and returns the loss.\n        \"\"\"\n        loss = None\n        if closure is not None:\n            with torch.enable_grad():\n                loss = closure()\n\n        for group in self.param_groups:\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                grad = p.grad\n                if grad.is_sparse:\n                    raise RuntimeError('Sparse gradients are not supported.')\n                amsgrad = group['amsgrad']\n\n                state = self.state[p]\n\n                # State initialization\n                if len(state) == 0:\n                    state['step'] = 0\n                    # Exponential moving average of gradient values\n                    state['exp_avg'] = torch.zeros_like(p)\n                    # Exponential moving average of squared gradient values\n                    state['exp_avg_sq'] = torch.zeros([]).to(state['exp_avg'].device)\n                    if amsgrad:\n                        # Maintains max of all exp. moving avg. of sq. grad. values\n                        state['max_exp_avg_sq'] = torch.zeros([]).to(state['exp_avg'].device)\n\n                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']\n                if amsgrad:\n                    max_exp_avg_sq = state['max_exp_avg_sq']","sourceCodeStart":67,"sourceCodeEnd":103,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/nvnovograd.py#L67-L103","documentation":"RuntimeError raised in Nvnovograd.step() when a parameter gradient is a sparse torch tensor; this optimizer implements only dense updates.","triggerScenarios":"Backward pass producing sparse grads (nn.Embedding(sparse=True)) followed by Nvnovograd.step().","commonSituations":"Training text/tabular models with sparse embeddings using timm's Nvnovograd, or porting a model from an optimizer that allowed sparsity.","solutions":["Create embeddings with sparse=False","Use a sparse-capable optimizer (e.g. SparseAdam) for sparse params","Split sparse and dense params into separate optimizers"],"exampleFix":"# before\nemb = nn.Embedding(vocab, dim, sparse=True)\nopt = Nvnovograd(model.parameters())\n\n# after\nemb = nn.Embedding(vocab, dim, sparse=False)\nopt = Nvnovograd(model.parameters())","handlingStrategy":"validation","validationCode":"assert all(p.grad is None or not p.grad.is_sparse for p in model.parameters())","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use dense gradients or a sparse-capable optimizer for embeddings"],"tags":["timm","nvnovograd","sparse-gradients","embedding"],"backgroundTag":"optimizer-sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}