huggingface/pytorch-image-models · error · ValueError

lr as a Tensor is not supported for capturable=False and for

Error message

lr as a Tensor is not supported for capturable=False and foreach=True

What it means

ADOPT's constructor rejects a tensor learning rate when foreach=True and capturable=False, because the multi-tensor foreach path without CUDA-graph capturability cannot read lr from a tensor without a device synchronization that breaks the fused implementation.

Source

Thrown at timm/optim/adopt.py:83

            self,
            params: ParamsT,
            lr: Union[float, Tensor] = 1e-3,
            betas: Tuple[float, float] = (0.9, 0.9999),
            eps: float = 1e-6,
            clip_exp: Optional[float] = 0.333,
            weight_decay: float = 0.0,
            decoupled: bool = False,
            corrected_weight_decay: bool = False,
            *,
            caution: bool = False,
            foreach: Optional[bool] = False,
            maximize: bool = False,
            capturable: bool = False,
            differentiable: bool = False,
    ):
        if isinstance(lr, Tensor):
            if foreach and not capturable:
                raise ValueError(
                    "lr as a Tensor is not supported for capturable=False and foreach=True"
                )
            if lr.numel() != 1:
                raise ValueError("Tensor lr must be 1-element")
        if not 0.0 <= lr:
            raise ValueError(f"Invalid learning rate: {lr}")
        if not 0.0 <= eps:
            raise ValueError(f"Invalid epsilon value: {eps}")
        if not 0.0 <= betas[0] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 0: {betas[0]}")
        if not 0.0 <= betas[1] < 1.0:
            raise ValueError(f"Invalid beta parameter at index 1: {betas[1]}")
        if not 0.0 <= weight_decay:
            raise ValueError(f"Invalid weight_decay value: {weight_decay}")

        defaults = dict(
            lr=lr,
            betas=betas,

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Pass capturable=True if you genuinely need a tensor lr with foreach
2. Use a plain float lr instead of a tensor
3. Set foreach=False to keep the tensor lr in the single-tensor path

Example fix

# before
opt = timm.optim.Adopt(model.parameters(), lr=torch.tensor(1e-3), foreach=True)
# after
opt = timm.optim.Adopt(model.parameters(), lr=1e-3)

if isinstance(lr, torch.Tensor):
    assert not (foreach and not capturable), 'tensor lr requires capturable=True or foreach=False'
    assert lr.numel() == 1

def valid_adopt_lr(lr, foreach: bool, capturable: bool) -> bool:
    if not isinstance(lr, torch.Tensor):
        return lr >= 0.0
    return lr.numel() == 1 and (capturable or not foreach)

When it happens

Trigger: Constructing timm.optim.Adopt(params, lr=torch.tensor(1e-3), foreach=True, capturable=False).

Common situations: Using a tensor lr for lr-scheduling tricks or per-device lr handling while leaving foreach at its default (which resolves to True when supported by the device).

Related errors

• Tensor lr must be 1-element
• Invalid learning rate: {lr}
• Invalid epsilon value: {eps}
• Invalid beta parameter at index 0: {betas[0]}
• Invalid beta parameter at index 1: {betas[1]}

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/f12dec87a46d0a68. Report an issue: GitHub.