{"record":{"id":"8c5361ae4756b493","repo":"labmlai/annotated_deep_learning_paper_implementations","slug":"invalid-learning-rate-lr","errorCode":null,"errorMessage":"Invalid learning rate: {lr}","messagePattern":"Invalid learning rate: (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"labml_nn/optimizers/__init__.py","lineNumber":88,"sourceCode":"class GenericAdaptiveOptimizer(Optimizer):\n    \"\"\"\n    ## Base class for *Adam* and extensions\n    \"\"\"\n\n    def __init__(self, params, defaults: Dict[str, Any], lr: float, betas: Tuple[float, float], eps: float):\n        \"\"\"\n        ### Initialize\n\n        * `params` is the collection of parameters or set of parameter groups.\n        * `defaults` a dictionary of default hyper-parameters\n        * `lr` is the learning rate, $\\alpha$\n        * `betas` is the tuple $(\\beta_1, \\beta_2)$\n        * `eps` is $\\epsilon$\n        \"\"\"\n\n        # Check the hyper-parameters\n        if not 0.0 <= lr:\n            raise ValueError(f\"Invalid learning rate: {lr}\")\n        if not 0.0 <= eps:\n            raise ValueError(f\"Invalid epsilon value: {eps}\")\n        if not 0.0 <= betas[0] < 1.0:\n            raise ValueError(f\"Invalid beta parameter at index 0: {betas[0]}\")\n        if not 0.0 <= betas[1] < 1.0:\n            raise ValueError(f\"Invalid beta parameter at index 1: {betas[1]}\")\n\n        # Add the hyper-parameters to the defaults\n        defaults.update(dict(lr=lr, betas=betas, eps=eps))\n        # Initialize the PyTorch optimizer.\n        # This will create parameter groups with the default hyper-parameters\n        super().__init__(params, defaults)\n\n    def init_state(self, state: Dict[str, any], group: Dict[str, any], param: nn.Parameter):\n        \"\"\"\n        ### Initialize state for a given parameter tensor\n\n        This should be overridden with code to initialize `state` for parameters `param`.","sourceCodeStart":70,"sourceCodeEnd":106,"githubUrl":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/33ab02281c2b928e6b32792909cc79cbdcfe1d6a/labml_nn/optimizers/__init__.py#L70-L106","documentation":"GenericAdaptiveOptimizer (base of Adam variants in labml_nn.optimizers) validates hyper-parameters in __init__. It requires lr >= 0 (negative learning rates are rejected; zero is allowed). A negative lr almost always indicates a sign error or a config parsing bug, so the constructor fails fast instead of producing a diverging training run.","triggerScenarios":"Constructing the optimizer with lr < 0, e.g. GenericAdaptiveOptimizer(params, lr=-1e-3) or any Adam subclass (Adam, AdamWarmup, etc.) with a negative lr from a config file or sweep tool.","commonSituations":"Learning-rate schedules evaluated at negative time/step values fed straight into the optimizer; YAML/JSON config typo (minus sign or bad exponent like 1e-4 vs -1e-4); hyperparameter search sampling an invalid range; argument order swapped (e.g. passing betas where lr goes).","solutions":["Print/inspect the lr value right before optimizer construction to find the sign or parse error","Fix the config: use a positive lr such as 1e-3 or 3e-4","If using a scheduler that can compute negative values, clamp: lr = max(0.0, lr)","In sweep configs, constrain the lr search space to positive values only"],"exampleFix":"# before\nopt = GenericAdaptiveOptimizer(model.parameters(), lr=-1e-3)\n\n# after\nopt = GenericAdaptiveOptimizer(model.parameters(), lr=1e-3)","handlingStrategy":"validation","validationCode":"lr = float(cfg['lr'])\nif not 0.0 <= lr:\n    raise ValueError(f'config lr must be >= 0, got {lr}')\nopt = GenericAdaptiveOptimizer(params, lr=lr)","typeGuard":"def valid_lr(lr: float) -> bool:\n    return isinstance(lr, (int, float)) and 0.0 <= lr < float('inf')","tryCatchPattern":"try:\n    opt = GenericAdaptiveOptimizer(params, lr=lr)\nexcept ValueError as e:\n    raise SystemExit(f'Bad optimizer config: {e}') from e","preventionTips":["Validate the full hyper-parameter dict once at startup","Constrain lr sweep spaces to positive log-uniform ranges","Never feed scheduler outputs directly as constructor lr without clamping"],"tags":["python","pytorch","optimizer","hyperparameter","validation"],"backgroundTag":"optimizer-hyperparameter-validation","analyzedSha":"33ab02281c2b928e6b32792909cc79cbdcfe1d6a","analyzedAt":"2026-08-25T10:30:27.743Z","schemaVersion":2},"datasetVersion":"2026-08-25T11:17:15.655Z"}