{"record":{"id":"29fb10b2564e69f9","repo":"huggingface/pytorch-image-models","slug":"rmsprop-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"RMSprop does not support sparse gradients","messagePattern":"RMSprop does not support sparse gradients","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/rmsprop_tf.py","lineNumber":118,"sourceCode":"    def step(self, closure=None):\n        \"\"\"Performs a single optimization step.\n\n        Arguments:\n            closure (callable, optional): A closure that reevaluates the model\n                and returns the loss.\n        \"\"\"\n        loss = None\n        if closure is not None:\n            with torch.enable_grad():\n                loss = closure()\n\n        for group in self.param_groups:\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                grad = p.grad\n                if grad.is_sparse:\n                    raise RuntimeError('RMSprop does not support sparse gradients')\n                state = self.state[p]\n\n                # State initialization\n                if len(state) == 0:\n                    state['step'] = _init_scalar(device='cpu')\n                    state['square_avg'] = torch.ones_like(p)  # PyTorch inits to zero\n                    if group['momentum'] > 0:\n                        state['momentum_buffer'] = torch.zeros_like(p)\n                    if group['centered']:\n                        state['grad_avg'] = torch.zeros_like(p)\n\n                square_avg = state['square_avg']\n                one_minus_alpha = 1. - group['alpha']\n\n                state['step'].add_(1)\n\n                if group['weight_decay'] != 0:\n                    if group['decoupled_decay']:","sourceCodeStart":100,"sourceCodeEnd":136,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/rmsprop_tf.py#L100-L136","documentation":"RuntimeError raised in RMSpropTF.step() when any parameter gradient is a sparse tensor. This TF-style RMSprop implementation performs dense-only in-place tensor math on square_avg/acc_grad buffers.","triggerScenarios":"Training a model with sparse gradients (e.g. nn.Embedding(sparse=True)) using timm.optim.RMSpropTF and calling step().","commonSituations":"Recommendation/search-ranking models with large embedding tables migrated to timm's RMSpropTF; configs reused from SGD-with-sparse setups.","solutions":["Set sparse=False on embedding layers","Use torch.optim.SparseAdam or torch.optim.SGD (which support sparse grads) for embedding params","Maintain separate optimizers for sparse vs dense parameter groups"],"exampleFix":"# before\nemb = nn.Embedding(num_items, 64, sparse=True)\nopt = RMSpropTF(model.parameters())\n\n# after\nemb = nn.Embedding(num_items, 64, sparse=False)\nopt = RMSpropTF(model.parameters())","handlingStrategy":"validation","validationCode":"assert all(p.grad is None or not p.grad.is_sparse for p in model.parameters())","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Avoid sparse=True with RMSpropTF","Use SparseAdam/SGD for sparse embedding groups"],"tags":["timm","rmsprop-tf","sparse-gradients","embedding"],"backgroundTag":"optimizer-sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}