geekcomputers/Python · error · ValueError

Invalid learning rate: {lr}

Error message

Invalid learning rate: {lr}

What it means

Raised by neuralforge's custom AdamW optimizer constructor when lr is negative. It mirrors torch.optim.AdamW's validation: learning rate must be non-negative. lr == 0 is accepted (only useful for schedules that will raise it later).

Source

Thrown at ML/src/python/neuralforge/optim/optimizers.py:8

import torch
from torch.optim.optimizer import Optimizer
import math

class AdamW(Optimizer):
    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0.01, amsgrad=False):
        if lr < 0.0:
            raise ValueError(f"Invalid learning rate: {lr}")
        if eps < 0.0:
            raise ValueError(f"Invalid epsilon value: {eps}")
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")
        
        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, amsgrad=amsgrad)
        super().__init__(params, defaults)
    
    def step(self, closure=None):
        loss = None
        if closure is not None:
            loss = closure()
        
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:

View on GitHub (pinned to 40f4cd2652)

Solutions

1. Inspect the lr value right before constructing the optimizer and log it
2. Clamp sweep/sample results: lr = max(lr, 0.0) or sample in log-space (e.g. 10**uniform(-5, -1))
3. Fix config typos or sentinel defaults like lr: -1
4. If lr legitimately starts at 0 for a scheduler, that is allowed; only negatives raise

Example fix

# before
import math
lr = 10 ** trial.suggest_float('log_lr', -5, 1)  # can exceed safe range / sign bugs
opt = AdamW(params, lr=-1e-3)  # ValueError

# after
lr = 10 ** trial.suggest_float('log_lr', -5, -3)
opt = AdamW(params, lr=max(lr, 0.0))

assert lr >= 0, f'lr must be >= 0, got {lr}'
opt = AdamW(params, lr=lr)

def is_valid_lr(lr) -> bool:
    return isinstance(lr, (int, float)) and lr >= 0

try:
    opt = AdamW(params, lr=lr)
except ValueError as e:
    if 'Invalid learning rate' in str(e):
        opt = AdamW(params, lr=abs(lr))  # or fall back to default 1e-3
    else:
        raise

When it happens

Trigger: Constructing AdamW(params, lr=-0.001); passing a value read from a config that defaulted to -1 as a placeholder; a learning-rate schedule or hyperparameter search proposing a negative value.

Common situations: Hyperparameter sweeps (optuna/raytune) sampling negative lr; config typos (negative sign); porting configs between libraries where lr semantics differ; deserialized checkpoint configs with sentinel values like -1.

Related errors

• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}
• AdamW does not support sparse gradients
• Please give a integer

AI-assisted analysis of geekcomputers/Python@40f4cd2652 (2026-08-27). Data as JSON: /api/errors/34315c818f0fee5a. Report an issue: GitHub.