huggingface/pytorch-image-models · error · ValueError

Weight decay {weight_decay} must be non-negative

Error message

Weight decay {weight_decay} must be non-negative

What it means

Weight decay must be >= 0 in MADGRAD. Negative weight decay would act as weight growth and, when coupled into the gradient, breaks the optimizer's convergence assumptions, so the constructor validates it.

Source

Thrown at timm/optim/madgrad.py:69

        eps (float):
            Term added to the denominator outside of the root operation to improve numerical stability. (default: 1e-6).
    """

    def __init__(
            self,
            params: _params_t,
            lr: float = 1e-2,
            momentum: float = 0.9,
            weight_decay: float = 0,
            eps: float = 1e-6,
            decoupled_decay: bool = False,
    ):
        if momentum < 0 or momentum >= 1:
            raise ValueError(f"Momentum {momentum} must be in the range [0,1]")
        if lr <= 0:
            raise ValueError(f"Learning rate {lr} must be positive")
        if weight_decay < 0:
            raise ValueError(f"Weight decay {weight_decay} must be non-negative")
        if eps < 0:
            raise ValueError(f"Eps must be non-negative")

        defaults = dict(
            lr=lr,
            eps=eps,
            momentum=momentum,
            weight_decay=weight_decay,
            decoupled_decay=decoupled_decay,
        )
        super().__init__(params, defaults)

    @property
    def supports_memory_efficient_fp16(self) -> bool:
        return False

    @property
    def supports_flat_params(self) -> bool:

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Use a non-negative weight_decay (0 to disable, e.g. 1e-4 for mild decay)
2. If you wanted weight growth, MADGRAD does not support it — remove the experiment

Example fix

# before
opt = MADGRAD(model.parameters(), lr=1e-3, weight_decay=-1e-4)
# after
opt = MADGRAD(model.parameters(), lr=1e-3, weight_decay=1e-4)

assert cfg.weight_decay >= 0, 'weight_decay must be non-negative'

def is_valid_wd(wd: float) -> bool:
    return isinstance(wd, (int, float)) and wd >= 0

When it happens

Trigger: Passing weight_decay=-1e-4 (e.g. a sign typo intended as L2 regularization) to MADGRAD.__init__.

Common situations: Sign errors ported from configs; experimenting with negative decay as a regularizer; config files where a minus sign was accidentally included.

Related errors

• Momentum {momentum} must be in the range [0,1]
• Learning rate {lr} must be positive
• Eps must be non-negative
• weight_decay option is not compatible with sparse gradients
• Invalid weight_decay value: {weight_decay}

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/e87c84c93c126f3f. Report an issue: GitHub.