{"record":{"id":"e87c84c93c126f3f","repo":"huggingface/pytorch-image-models","slug":"weight-decay-weight-decay-must-be-non-negative","errorCode":null,"errorMessage":"Weight decay {weight_decay} must be non-negative","messagePattern":"Weight decay (.+?) must be non-negative","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"timm/optim/madgrad.py","lineNumber":69,"sourceCode":"        eps (float):\n            Term added to the denominator outside of the root operation to improve numerical stability. (default: 1e-6).\n    \"\"\"\n\n    def __init__(\n            self,\n            params: _params_t,\n            lr: float = 1e-2,\n            momentum: float = 0.9,\n            weight_decay: float = 0,\n            eps: float = 1e-6,\n            decoupled_decay: bool = False,\n    ):\n        if momentum < 0 or momentum >= 1:\n            raise ValueError(f\"Momentum {momentum} must be in the range [0,1]\")\n        if lr <= 0:\n            raise ValueError(f\"Learning rate {lr} must be positive\")\n        if weight_decay < 0:\n            raise ValueError(f\"Weight decay {weight_decay} must be non-negative\")\n        if eps < 0:\n            raise ValueError(f\"Eps must be non-negative\")\n\n        defaults = dict(\n            lr=lr,\n            eps=eps,\n            momentum=momentum,\n            weight_decay=weight_decay,\n            decoupled_decay=decoupled_decay,\n        )\n        super().__init__(params, defaults)\n\n    @property\n    def supports_memory_efficient_fp16(self) -> bool:\n        return False\n\n    @property\n    def supports_flat_params(self) -> bool:","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/optim/madgrad.py#L51-L87","documentation":"Weight decay must be >= 0 in MADGRAD. Negative weight decay would act as weight growth and, when coupled into the gradient, breaks the optimizer's convergence assumptions, so the constructor validates it.","triggerScenarios":"Passing weight_decay=-1e-4 (e.g. a sign typo intended as L2 regularization) to MADGRAD.__init__.","commonSituations":"Sign errors ported from configs; experimenting with negative decay as a regularizer; config files where a minus sign was accidentally included.","solutions":["Use a non-negative weight_decay (0 to disable, e.g. 1e-4 for mild decay)","If you wanted weight growth, MADGRAD does not support it — remove the experiment"],"exampleFix":"# before\nopt = MADGRAD(model.parameters(), lr=1e-3, weight_decay=-1e-4)\n# after\nopt = MADGRAD(model.parameters(), lr=1e-3, weight_decay=1e-4)","handlingStrategy":"validation","validationCode":"assert cfg.weight_decay >= 0, 'weight_decay must be non-negative'","typeGuard":"def is_valid_wd(wd: float) -> bool:\n    return isinstance(wd, (int, float)) and wd >= 0","tryCatchPattern":null,"preventionTips":["Validate weight_decay sign in config schemas","Use 0 to disable decay rather than negative values"],"tags":["optimizer","madgrad","weight-decay","hyperparameter-validation"],"backgroundTag":"hyperparameter-out-of-range","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}