labmlai/annotated_deep_learning_paper_implementations · error · ValueError

Invalid epsilon value: {eps}

Error message

Invalid epsilon value: {eps}

What it means

GenericAdaptiveOptimizer's constructor requires eps >= 0 because epsilon is added to denominators for numerical stability. A negative eps would corrupt the adaptive scaling, so the constructor raises ValueError for any negative epsilon.

Source

Thrown at labml_nn/optimizers/__init__.py:90

    ## Base class for *Adam* and extensions
    """

    def __init__(self, params, defaults: Dict[str, Any], lr: float, betas: Tuple[float, float], eps: float):
        """
        ### Initialize

        * `params` is the collection of parameters or set of parameter groups.
        * `defaults` a dictionary of default hyper-parameters
        * `lr` is the learning rate, $\alpha$
        * `betas` is the tuple $(\beta_1, \beta_2)$
        * `eps` is $\epsilon$
        """

        # Check the hyper-parameters
        if not 0.0 <= lr:
            raise ValueError(f"Invalid learning rate: {lr}")
        if not 0.0 <= eps:
            raise ValueError(f"Invalid epsilon value: {eps}")
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")

        # Add the hyper-parameters to the defaults
        defaults.update(dict(lr=lr, betas=betas, eps=eps))
        # Initialize the PyTorch optimizer.
        # This will create parameter groups with the default hyper-parameters
        super().__init__(params, defaults)

    def init_state(self, state: Dict[str, any], group: Dict[str, any], param: nn.Parameter):
        """
        ### Initialize state for a given parameter tensor

        This should be overridden with code to initialize `state` for parameters `param`.
        `group` is the parameter group dictionary to which `param` belongs.
        """

View on GitHub (pinned to 33ab02281c)

Solutions

1. Check the eps value in your config; use a positive epsilon such as 1e-8
2. Constrain sweep ranges for eps to positive values
3. Add a startup assertion on all hyper-parameters before building the optimizer

Example fix

# before
opt = GenericAdaptiveOptimizer(model.parameters(), lr=1e-3, eps=-1e-8)

# after
opt = GenericAdaptiveOptimizer(model.parameters(), lr=1e-3, eps=1e-8)

eps = float(cfg['eps'])
if not 0.0 <= eps:
    raise ValueError(f'config eps must be >= 0, got {eps}')

def valid_eps(eps: float) -> bool:
    return isinstance(eps, (int, float)) and 0.0 <= eps < float('inf')

try:
    opt = GenericAdaptiveOptimizer(params, lr=lr, eps=eps)
except ValueError as e:
    raise SystemExit(f'Bad optimizer config: {e}') from e

When it happens

Trigger: Constructing the optimizer with eps < 0, e.g. GenericAdaptiveOptimizer(params, eps=-1e-8), or any labml-nn Adam variant with a negative epsilon from config/sweep.

Common situations: Config typo on the exponent (1e-8 mistyped as -1e-8); hyperparameter search with a symmetric range around zero; copy-paste from a paper table where eps was listed with a dash; positional-argument mixups.

Related errors

• Invalid learning rate: {lr}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}
• Invalid weight_decay value: {weight_decay}
• GenericAdaptiveOptimizer does not support sparse gradients,

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/8321936b732da7df. Report an issue: GitHub.