labmlai/annotated_deep_learning_paper_implementations · error · ValueError

Invalid learning rate: {lr}

Error message

Invalid learning rate: {lr}

What it means

GenericAdaptiveOptimizer (base of Adam variants in labml_nn.optimizers) validates hyper-parameters in __init__. It requires lr >= 0 (negative learning rates are rejected; zero is allowed). A negative lr almost always indicates a sign error or a config parsing bug, so the constructor fails fast instead of producing a diverging training run.

Source

Thrown at labml_nn/optimizers/__init__.py:88

class GenericAdaptiveOptimizer(Optimizer):
    """
    ## Base class for *Adam* and extensions
    """

    def __init__(self, params, defaults: Dict[str, Any], lr: float, betas: Tuple[float, float], eps: float):
        """
        ### Initialize

        * `params` is the collection of parameters or set of parameter groups.
        * `defaults` a dictionary of default hyper-parameters
        * `lr` is the learning rate, $\alpha$
        * `betas` is the tuple $(\beta_1, \beta_2)$
        * `eps` is $\epsilon$
        """

        # Check the hyper-parameters
        if not 0.0 <= lr:
            raise ValueError(f"Invalid learning rate: {lr}")
        if not 0.0 <= eps:
            raise ValueError(f"Invalid epsilon value: {eps}")
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")

        # Add the hyper-parameters to the defaults
        defaults.update(dict(lr=lr, betas=betas, eps=eps))
        # Initialize the PyTorch optimizer.
        # This will create parameter groups with the default hyper-parameters
        super().__init__(params, defaults)

    def init_state(self, state: Dict[str, any], group: Dict[str, any], param: nn.Parameter):
        """
        ### Initialize state for a given parameter tensor

        This should be overridden with code to initialize `state` for parameters `param`.

View on GitHub (pinned to 33ab02281c)

Solutions

1. Print/inspect the lr value right before optimizer construction to find the sign or parse error
2. Fix the config: use a positive lr such as 1e-3 or 3e-4
3. If using a scheduler that can compute negative values, clamp: lr = max(0.0, lr)
4. In sweep configs, constrain the lr search space to positive values only

Example fix

# before
opt = GenericAdaptiveOptimizer(model.parameters(), lr=-1e-3)

# after
opt = GenericAdaptiveOptimizer(model.parameters(), lr=1e-3)

lr = float(cfg['lr'])
if not 0.0 <= lr:
    raise ValueError(f'config lr must be >= 0, got {lr}')
opt = GenericAdaptiveOptimizer(params, lr=lr)

def valid_lr(lr: float) -> bool:
    return isinstance(lr, (int, float)) and 0.0 <= lr < float('inf')

try:
    opt = GenericAdaptiveOptimizer(params, lr=lr)
except ValueError as e:
    raise SystemExit(f'Bad optimizer config: {e}') from e

When it happens

Trigger: Constructing the optimizer with lr < 0, e.g. GenericAdaptiveOptimizer(params, lr=-1e-3) or any Adam subclass (Adam, AdamWarmup, etc.) with a negative lr from a config file or sweep tool.

Common situations: Learning-rate schedules evaluated at negative time/step values fed straight into the optimizer; YAML/JSON config typo (minus sign or bad exponent like 1e-4 vs -1e-4); hyperparameter search sampling an invalid range; argument order swapped (e.g. passing betas where lr goes).

Related errors

• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}
• Invalid weight_decay value: {weight_decay}
• GenericAdaptiveOptimizer does not support sparse gradients,

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/8c5361ae4756b493. Report an issue: GitHub.