huggingface/pytorch-image-models · error · ValueError

Invalid weight_decay value: {weight_decay}

Error message

Invalid weight_decay value: {weight_decay}

What it means

Raised by ADOPT's constructor when weight_decay is negative. ADOPT decouples weight decay as an added non-negative penalty; negative decay would grow weights and is rejected.

Source

Thrown at timm/optim/adopt.py:97

            differentiable: bool = False,
    ):
        if isinstance(lr, Tensor):
            if foreach and not capturable:
                raise ValueError(
                    "lr as a Tensor is not supported for capturable=False and foreach=True"
                )
            if lr.numel() != 1:
                raise ValueError("Tensor lr must be 1-element")
        if not 0.0 <= lr:
            raise ValueError(f"Invalid learning rate: {lr}")
        if not 0.0 <= eps:
            raise ValueError(f"Invalid epsilon value: {eps}")
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")
        if not 0.0 <= weight_decay:
            raise ValueError(f"Invalid weight_decay value: {weight_decay}")

        defaults = dict(
            lr=lr,
            betas=betas,
            eps=eps,
            weight_decay=weight_decay,
            clip_exp=clip_exp,
            decoupled=decoupled,
            corrected_weight_decay=corrected_weight_decay,
            caution=caution,
            maximize=maximize,
            foreach=foreach,
            capturable=capturable,
            differentiable=differentiable,
        )
        super().__init__(params, defaults)

    def __setstate__(self, state):

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Use a non-negative weight_decay, typically 0.0 to 0.1 (e.g. 0.02)
2. If you wanted no decay, pass weight_decay=0.0 explicitly

Example fix

# before
opt = timm.optim.Adopt(model.parameters(), lr=1e-3, weight_decay=-0.02)
# after
opt = timm.optim.Adopt(model.parameters(), lr=1e-3, weight_decay=0.02)

assert weight_decay >= 0.0, f'weight_decay must be >= 0, got {weight_decay}'

def valid_weight_decay(wd) -> bool:
    return isinstance(wd, (int, float)) and wd >= 0.0

When it happens

Trigger: Constructing timm.optim.Adopt(params, weight_decay=-1e-4).

Common situations: Sign typo in config, or confusing weight decay with a norm-growth regularizer; sweeps exploring negative decay values.

Related errors

• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}
• Invalid beta parameter at index 0: {}

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/1050598b4e5f564b. Report an issue: GitHub.