labmlai/annotated_deep_learning_paper_implementations · error · ValueError

Invalid weight_decay value: {weight_decay}

Error message

Invalid weight_decay value: {weight_decay}

What it means

The WeightDecay helper used by labml-nn optimizers (their AdamW-style decoupled decay) validates that weight_decay >= 0. Negative weight decay would grow weights instead of shrinking them and is never meaningful, so __init__ raises ValueError immediately.

Source

Thrown at labml_nn/optimizers/__init__.py:186

    """
    ## L2 Weight decay
    """

    def __init__(self, weight_decay: float = 0., weight_decouple: bool = True, absolute: bool = False):
        """
        ### Initialize weight decay

        * `weight_decay` is the decay coefficient
        * `weight_decouple` is a flag indicating whether to add the weight decay to the gradient or directly
        decay from the parameter. If added to the  gradient it will go through the normal optimizer update.
        * `absolute` this flag indicates whether the weight decay coefficient is absolute. This is applicable
        when the decay is performed directly on the parameter. If this is false the actual decay is
        `weight_decay`
        * `learning_rate`.
        """
        # Check hyper-parameters
        if not 0.0 <= weight_decay:
            raise ValueError(f"Invalid weight_decay value: {weight_decay}")

        self.absolute = absolute
        self.weight_decouple = weight_decouple
        self.weight_decay = weight_decay

    def defaults(self):
        """
        Return defaults for parameter groups
        """
        return dict(weight_decay=self.weight_decay)

    def __call__(self, param: torch.nn.Parameter, grad: torch.Tensor, group: Dict[str, any]):
        """
        ### Perform weight decay and return the gradient
        """

        # If we are doing the decay on the parameter directly
        if self.weight_decouple:

View on GitHub (pinned to 33ab02281c)

Solutions

1. Use a non-negative weight_decay such as 0.01 or 1e-2 (or 0.0 to disable decay)
2. Constrain the sweep range for weight_decay to [0, upper]
3. Validate config values before constructing the optimizer

Example fix

# before
opt = GenericAdaptiveOptimizer(params, lr=1e-3, weight_decay=-0.01)

# after
opt = GenericAdaptiveOptimizer(params, lr=1e-3, weight_decay=0.01)

wd = float(cfg['weight_decay'])
if not 0.0 <= wd:
    raise ValueError(f'weight_decay must be >= 0, got {wd}')

def valid_weight_decay(wd: float) -> bool:
    return isinstance(wd, (int, float)) and 0.0 <= wd < float('inf')

try:
    opt = GenericAdaptiveOptimizer(params, lr=lr, weight_decay=wd)
except ValueError as e:
    raise SystemExit(f'Bad optimizer config: {e}') from e

When it happens

Trigger: Constructing the optimizer/WeightDecay with weight_decay < 0, e.g. GenericAdaptiveOptimizer(params, lr=1e-3, weight_decay=-0.01) or a sweep/config supplying a negative decay coefficient.

Common situations: Sign typo in config (-0.01 vs 0.01); sweeps sampling weight_decay symmetric around zero; confusing weight_decay with a gain term; YAML parsing issues.

Related errors

• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}
• GenericAdaptiveOptimizer does not support sparse gradients,

AI-assisted analysis of labmlai/annotated_deep_learning_paper_implementations@33ab02281c (2026-08-25). Data as JSON: /api/errors/0e40f3b6d2022934. Report an issue: GitHub.