labmlai/annotated_deep_learning_paper_implementations · error · ValueError

Invalid beta parameter at index 0: {betas[0]}

Error message

Invalid beta parameter at index 0: {betas[0]}

What it means

GenericAdaptiveOptimizer requires each momentum coefficient beta to satisfy 0 <= beta < 1. beta[0] (beta1) is the exponential decay rate for the first-moment (gradient moving average) estimate; values >= 1.0 or negative values make the moving average mathematically invalid (non-decaying or diverging), so the constructor rejects them.

Source

Thrown at labml_nn/optimizers/__init__.py:92

    def __init__(self, params, defaults: Dict[str, Any], lr: float, betas: Tuple[float, float], eps: float):
        """
        ### Initialize

        * `params` is the collection of parameters or set of parameter groups.
        * `defaults` a dictionary of default hyper-parameters
        * `lr` is the learning rate, $\alpha$
        * `betas` is the tuple $(\beta_1, \beta_2)$
        * `eps` is $\epsilon$
        """

        # Check the hyper-parameters
        if not 0.0 <= lr:
            raise ValueError(f"Invalid learning rate: {lr}")
        if not 0.0 <= eps:
            raise ValueError(f"Invalid epsilon value: {eps}")
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")

        # Add the hyper-parameters to the defaults
        defaults.update(dict(lr=lr, betas=betas, eps=eps))
        # Initialize the PyTorch optimizer.
        # This will create parameter groups with the default hyper-parameters
        super().__init__(params, defaults)

    def init_state(self, state: Dict[str, any], group: Dict[str, any], param: nn.Parameter):
        """
        ### Initialize state for a given parameter tensor

        This should be overridden with code to initialize `state` for parameters `param`.
        `group` is the parameter group dictionary to which `param` belongs.
        """
        pass

View on GitHub (pinned to 33ab02281c)

Solutions

1. Use a valid beta1 in [0, 1), typically 0.9
2. Fix the config/sweep range for beta1 to [0.0, 1.0) exclusive
3. Validate betas before construction: assert all(0.0 <= b < 1.0 for b in betas)

Example fix

# before: beta1 = 1.0 -> raises
opt = GenericAdaptiveOptimizer(params, lr=1e-3, betas=(1.0, 0.999))

# after
opt = GenericAdaptiveOptimizer(params, lr=1e-3, betas=(0.9, 0.999))

beta1, beta2 = cfg['betas']
if not (0.0 <= beta1 < 1.0 and 0.0 <= beta2 < 1.0):
    raise ValueError(f'betas must be in [0, 1), got {(beta1, beta2)}')

def valid_betas(betas: tuple) -> bool:
    return (len(betas) == 2
            and all(isinstance(b, (int, float)) for b in betas)
            and all(0.0 <= b < 1.0 for b in betas))

try:
    opt = GenericAdaptiveOptimizer(params, lr=lr, betas=betas)
except ValueError as e:
    raise SystemExit(f'Bad optimizer config: {e}') from e

When it happens

Trigger: Constructing the optimizer with betas[0] outside [0, 1), e.g. betas=(1.0, 0.999), betas=(-0.9, 0.999), or betas=(1.1, 0.98).

Common situations: Copy-paste where 0.9 becomes 1.0 or 9.0; hyperparameter sweep sampling outside the valid range; misreading a paper's beta1; accidentally passing a tuple with swapped or extra elements.

Related errors

• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}
• Invalid beta parameter at index 1: {betas[1]}
• Invalid weight_decay value: {weight_decay}
• GenericAdaptiveOptimizer does not support sparse gradients,

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/ce3add3387557a61. Report an issue: GitHub.