{"record":{"id":"291c7434ba061d13","repo":"huggingface/pytorch-image-models","slug":"muon-does-not-support-sparse-gradients","errorCode":null,"errorMessage":"Muon does not support sparse gradients","messagePattern":"Muon does not support sparse gradients","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"timm/optim/muon.py","lineNumber":825,"sourceCode":"            muon_params = []\n            muon_grads = []\n            muon_momentum_bufs = []\n            # Additional state for adamuon mode\n            muon_exp_avg_sqs = []\n            muon_state_steps = []\n\n            adamw_params = []\n            adamw_grads = []\n            adamw_exp_avgs = []\n            adamw_exp_avg_sqs = []\n            adamw_state_steps = []\n\n            for p in group[\"params\"]:\n                if p.grad is None:\n                    continue\n\n                if p.grad.is_sparse:\n                    raise RuntimeError(\"Muon does not support sparse gradients\")\n\n                state = self.state[p]\n\n                # Determine routing on first encounter (cache in state)\n                if \"use_muon\" not in state:\n                    # Check explicit flags first (support both 'use_fallback' and 'use_muon' for compatibility)\n                    reason = None\n                    if group.get(\"use_fallback\", False):\n                        # use_fallback=True means use AdamW (use_muon=False)\n                        state[\"use_muon\"] = False\n                        if verbose:\n                            reason = \"use_fallback_flag\"\n                    elif \"use_muon\" in group:\n                        # Explicit use_muon flag for compatibility with other Muon implementations\n                        state[\"use_muon\"] = group[\"use_muon\"]\n                        if verbose:\n                            reason = \"use_muon_flag\"\n                    else:","sourceCodeStart":807,"sourceCodeEnd":843,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/muon.py#L807-L843","documentation":"Muon's update relies on dense matrix operations (Newton–Schulz orthogonalization of the gradient), so step() raises immediately if any parameter's gradient is a sparse tensor.","triggerScenarios":"optimizer.step() with a parameter whose .grad is sparse — typically nn.Embedding(sparse=True) — while using the Muon optimizer.","commonSituations":"Applying Muon to a whole model that includes sparse embeddings (language models, recsys); reusing a sparse training pipeline with a new Muon config.","solutions":["Remove sparse=True from embeddings so gradients are dense","Put sparse-gradient params in a separate param group optimized by torch.optim.SparseAdam or SGD","Exclude embeddings from the Muon optimizer entirely"],"exampleFix":"# before\nemb = nn.Embedding(vocab, dim, sparse=True)\nopt = Muon(model.parameters())\n# after\nemb = nn.Embedding(vocab, dim)\nopt = Muon(model.parameters())","handlingStrategy":"type-guard","validationCode":"assert all(p.grad is None or not p.grad.is_sparse for p in params), 'Muon cannot step on sparse gradients'","typeGuard":"def has_sparse_grads(params) -> bool:\n    return any(p.grad is not None and p.grad.is_sparse for p in params)","tryCatchPattern":null,"preventionTips":["Drop sparse=True from embeddings in Muon training","Use a separate SparseAdam group for sparse params"],"tags":["optimizer","muon","sparse-gradients"],"backgroundTag":"sparse-gradient-unsupported","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}