huggingface/pytorch-image-models · error · ValueError

Learning rate {lr} must be positive

Error message

Learning rate {lr} must be positive

What it means

MADGRAD requires a strictly positive learning rate because its update rule divides by quantities derived from lr; a zero or negative lr is meaningless and is rejected at construction time.

Source

Thrown at timm/optim/madgrad.py:67

        weight_decay (float):
            Weight decay, i.e. a L2 penalty (default: 0).
        eps (float):
            Term added to the denominator outside of the root operation to improve numerical stability. (default: 1e-6).
    """

    def __init__(
            self,
            params: _params_t,
            lr: float = 1e-2,
            momentum: float = 0.9,
            weight_decay: float = 0,
            eps: float = 1e-6,
            decoupled_decay: bool = False,
    ):
        if momentum < 0 or momentum >= 1:
            raise ValueError(f"Momentum {momentum} must be in the range [0,1]")
        if lr <= 0:
            raise ValueError(f"Learning rate {lr} must be positive")
        if weight_decay < 0:
            raise ValueError(f"Weight decay {weight_decay} must be non-negative")
        if eps < 0:
            raise ValueError(f"Eps must be non-negative")

        defaults = dict(
            lr=lr,
            eps=eps,
            momentum=momentum,
            weight_decay=weight_decay,
            decoupled_decay=decoupled_decay,
        )
        super().__init__(params, defaults)

    @property
    def supports_memory_efficient_fp16(self) -> bool:
        return False

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Pass a positive lr such as 1e-3 (MADGRAD's typical default)
2. Fix the config/CLI plumbing that produced 0 or a negative value
3. Exclude lr<=0 from sweep grids

Example fix

# before
opt = MADGRAD(model.parameters(), lr=0)
# after
opt = MADGRAD(model.parameters(), lr=1e-3)

assert cfg.lr > 0, 'lr must be positive for MADGRAD'

def is_valid_lr(lr: float) -> bool:
    return isinstance(lr, (int, float)) and lr > 0

When it happens

Trigger: Calling MADGRAD(params, lr=0) or lr=-0.1, e.g. when lr comes from a sweep that includes 0, or when a config key is missing and defaults to 0.

Common situations: Hyperparameter searches that probe lr=0; misparsed CLI args; a scheduler/JSON config typo producing 0 or a negative float.

Related errors

• Momentum {momentum} must be in the range [0,1]
• Weight decay {weight_decay} must be non-negative
• Eps must be non-negative
• Invalid learning rate: {}
• momentum != 0 is not compatible with sparse gradients

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/1fd4119ef54f2b49. Report an issue: GitHub.