rohitg00/ai-engineering-from-scratch · error · ValueError

inverse-sqrt warmup requires warmup_steps > 0

Error message

inverse-sqrt warmup requires warmup_steps > 0

What it means

Error "inverse-sqrt warmup requires warmup_steps > 0" thrown in rohitg00/ai-engineering-from-scratch.

Source

Thrown at phases/19-capstone-projects/44-cosine-lr-warmup/code/main.py:282

        if step >= self.warmup_steps:
            return self.lr_max
        return self.lr_max * (step / self.warmup_steps)


@dataclass
class InverseSqrtWarmup:
    """Linear warmup followed by an inverse-square-root decay.

    Decays as lr_max * sqrt(warmup_steps / step) for step > warmup_steps. Used
    historically in transformer training and useful as a comparison baseline.
    """

    warmup_steps: int
    lr_max: float

    def __post_init__(self) -> None:
        if self.warmup_steps <= 0:
            raise ValueError("inverse-sqrt warmup requires warmup_steps > 0")
        if self.lr_max <= 0:
            raise ValueError("lr_max must be positive")

    def lr(self, step: int) -> float:
        if step < 0:
            raise ValueError(f"step must be non-negative, got {step}")
        if step <= self.warmup_steps:
            return self.lr_max * (step / self.warmup_steps)
        return self.lr_max * math.sqrt(self.warmup_steps / step)


@dataclass
class EWMA:
    """Exponentially weighted moving average of a scalar, useful for grad-norm smoothing."""

    beta: float
    value: float = 0.0
    initialized: bool = False

View on GitHub (pinned to 39ea8a1c6d)

When it happens

Trigger: Thrown at phases/19-capstone-projects/44-cosine-lr-warmup/code/main.py:282 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of rohitg00/ai-engineering-from-scratch@39ea8a1c6d (2026-08-26). Data as JSON: /api/errors/c44490f88df164b4. Report an issue: GitHub.