{"record":{"id":"0e40f3b6d2022934","repo":"labmlai/annotated_deep_learning_paper_implementations","slug":"invalid-weight-decay-value-weight-decay","errorCode":null,"errorMessage":"Invalid weight_decay value: {weight_decay}","messagePattern":"Invalid weight_decay value: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"labml_nn/optimizers/__init__.py","lineNumber":186,"sourceCode":"    \"\"\"\n    ## L2 Weight decay\n    \"\"\"\n\n    def __init__(self, weight_decay: float = 0., weight_decouple: bool = True, absolute: bool = False):\n        \"\"\"\n        ### Initialize weight decay\n\n        * `weight_decay` is the decay coefficient\n        * `weight_decouple` is a flag indicating whether to add the weight decay to the gradient or directly\n        decay from the parameter. If added to the  gradient it will go through the normal optimizer update.\n        * `absolute` this flag indicates whether the weight decay coefficient is absolute. This is applicable\n        when the decay is performed directly on the parameter. If this is false the actual decay is\n        `weight_decay`\n        * `learning_rate`.\n        \"\"\"\n        # Check hyper-parameters\n        if not 0.0 <= weight_decay:\n            raise ValueError(f\"Invalid weight_decay value: {weight_decay}\")\n\n        self.absolute = absolute\n        self.weight_decouple = weight_decouple\n        self.weight_decay = weight_decay\n\n    def defaults(self):\n        \"\"\"\n        Return defaults for parameter groups\n        \"\"\"\n        return dict(weight_decay=self.weight_decay)\n\n    def __call__(self, param: torch.nn.Parameter, grad: torch.Tensor, group: Dict[str, any]):\n        \"\"\"\n        ### Perform weight decay and return the gradient\n        \"\"\"\n\n        # If we are doing the decay on the parameter directly\n        if self.weight_decouple:","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/33ab02281c2b928e6b32792909cc79cbdcfe1d6a/labml_nn/optimizers/__init__.py#L168-L204","documentation":"The WeightDecay helper used by labml-nn optimizers (their AdamW-style decoupled decay) validates that weight_decay >= 0. Negative weight decay would grow weights instead of shrinking them and is never meaningful, so __init__ raises ValueError immediately.","triggerScenarios":"Constructing the optimizer/WeightDecay with weight_decay < 0, e.g. GenericAdaptiveOptimizer(params, lr=1e-3, weight_decay=-0.01) or a sweep/config supplying a negative decay coefficient.","commonSituations":"Sign typo in config (-0.01 vs 0.01); sweeps sampling weight_decay symmetric around zero; confusing weight_decay with a gain term; YAML parsing issues.","solutions":["Use a non-negative weight_decay such as 0.01 or 1e-2 (or 0.0 to disable decay)","Constrain the sweep range for weight_decay to [0, upper]","Validate config values before constructing the optimizer"],"exampleFix":"# before\nopt = GenericAdaptiveOptimizer(params, lr=1e-3, weight_decay=-0.01)\n\n# after\nopt = GenericAdaptiveOptimizer(params, lr=1e-3, weight_decay=0.01)","handlingStrategy":"validation","validationCode":"wd = float(cfg['weight_decay'])\nif not 0.0 <= wd:\n    raise ValueError(f'weight_decay must be >= 0, got {wd}')","typeGuard":"def valid_weight_decay(wd: float) -> bool:\n    return isinstance(wd, (int, float)) and 0.0 <= wd < float('inf')","tryCatchPattern":"try:\n    opt = GenericAdaptiveOptimizer(params, lr=lr, weight_decay=wd)\nexcept ValueError as e:\n    raise SystemExit(f'Bad optimizer config: {e}') from e","preventionTips":["Use 0.0 to disable decay; typical valid values are 0.01–0.1","Constrain sweep ranges for weight_decay to non-negative values","Validate the whole optimizer config dict once at startup"],"tags":["python","pytorch","optimizer","hyperparameter","weight-decay","validation"],"backgroundTag":"optimizer-hyperparameter-validation","analyzedSha":"33ab02281c2b928e6b32792909cc79cbdcfe1d6a","analyzedAt":"2026-08-25T10:30:27.743Z","schemaVersion":2},"datasetVersion":"2026-08-25T11:17:15.655Z"}