labmlai/annotated_deep_learning_paper_implementations · error · ValueError

Invalid beta parameter at index 1: {betas[1]}

Error message

Invalid beta parameter at index 1: {betas[1]}

What it means

GenericAdaptiveOptimizer requires 0 <= betas[1] < 1. beta[1] (beta2) is the decay rate for the second-moment (squared-gradient moving average) estimate used in the adaptive denominator. beta2 >= 1 or negative values break convergence, so the constructor raises ValueError for them.

Source

Thrown at labml_nn/optimizers/__init__.py:94

        """
        ### Initialize

        * `params` is the collection of parameters or set of parameter groups.
        * `defaults` a dictionary of default hyper-parameters
        * `lr` is the learning rate, $\alpha$
        * `betas` is the tuple $(\beta_1, \beta_2)$
        * `eps` is $\epsilon$
        """

        # Check the hyper-parameters
        if not 0.0 <= lr:
            raise ValueError(f"Invalid learning rate: {lr}")
        if not 0.0 <= eps:
            raise ValueError(f"Invalid epsilon value: {eps}")
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")

        # Add the hyper-parameters to the defaults
        defaults.update(dict(lr=lr, betas=betas, eps=eps))
        # Initialize the PyTorch optimizer.
        # This will create parameter groups with the default hyper-parameters
        super().__init__(params, defaults)

    def init_state(self, state: Dict[str, any], group: Dict[str, any], param: nn.Parameter):
        """
        ### Initialize state for a given parameter tensor

        This should be overridden with code to initialize `state` for parameters `param`.
        `group` is the parameter group dictionary to which `param` belongs.
        """
        pass

    def step_param(self, state: Dict[str, any], group: Dict[str, any], grad: torch.Tensor, param: torch.Tensor):
        """

View on GitHub (pinned to 33ab02281c)

Solutions

1. Use a valid beta2 in [0, 1), typically 0.999 or 0.98
2. Restrict the sweep/config range for beta2 to values strictly below 1.0
3. Add a pre-construction check on the betas tuple

Example fix

# before: beta2 = 1.0 -> raises
opt = GenericAdaptiveOptimizer(params, lr=1e-3, betas=(0.9, 1.0))

# after
opt = GenericAdaptiveOptimizer(params, lr=1e-3, betas=(0.9, 0.999))

beta2 = cfg['betas'][1]
if not 0.0 <= beta2 < 1.0:
    raise ValueError(f'beta2 must be in [0, 1), got {beta2}')

def valid_beta2(beta2: float) -> bool:
    return isinstance(beta2, (int, float)) and 0.0 <= beta2 < 1.0

try:
    opt = GenericAdaptiveOptimizer(params, lr=lr, betas=(beta1, beta2))
except ValueError as e:
    raise SystemExit(f'Bad optimizer config: {e}') from e

When it happens

Trigger: Constructing the optimizer with betas[1] outside [0, 1), e.g. betas=(0.9, 1.0), betas=(0.9, 0.9999) is fine but betas=(0.9, -0.999) or (0.9, 1.1) raise.

Common situations: Typo turning 0.999 into 1.00 or 9.99; sweeps sampling beta2 over [0, 2]; configs migrated from optimizers that allowed beta2 = 1 (degenerate); YAML parsing 0.999 as a string.

Related errors

• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid weight_decay value: {weight_decay}
• GenericAdaptiveOptimizer does not support sparse gradients,

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/bef63e6be3ba305f. Report an issue: GitHub.