Lightning-AI/pytorch-lightning · error · ValueError

Expected the value to increase, last: {last}, current: {x}

Error message

Expected the value to increase, last: {last}, current: {x}

What it means

_Monotonic increasing list (used to track measured elapsed time intervals for throughput estimation) enforces strictly increasing values. append() raises ValueError when the new value is <= the last recorded value, because time going backwards would corrupt the extrapolation.

Source

Thrown at src/lightning/fabric/utilities/throughput.py:710

class _MonotonicWindow(list[T]):
    """Custom fixed size list that only supports right-append and ensures that all values increase monotonically."""

    def __init__(self, maxlen: int) -> None:
        super().__init__()
        self.maxlen = maxlen

    @property
    def last(self) -> Optional[T]:
        if len(self) > 0:
            return self[-1]
        return None

    @override
    def append(self, x: T) -> None:
        last = self.last
        if last is not None and last >= x:
            raise ValueError(f"Expected the value to increase, last: {last}, current: {x}")
        list.append(self, x)
        # truncate excess
        if len(self) > self.maxlen:
            del self[0]

    @override
    def __setitem__(self, key: Any, value: Any) -> None:
        # assigning is not implemented since we don't use it. it could be by checking all previous values
        raise NotImplementedError("__setitem__ is not supported")

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Ensure each append uses a freshly computed elapsed interval (time.perf_counter() delta) with enough resolution
2. Reset the ThroughputMonitor / timer state when resuming a run or re-running measurement in the same process
3. In tests or mocked environments, inject strictly increasing clock values

Example fix

# before
intervals.append(0.0)  # repeated value from coarse clock

# after
import time
now = time.perf_counter()
intervals.append(max(now - last_timestamp, 1e-9))
last_timestamp = now

last = intervals.last
if last is not None and value <= last:
    value = last + 1e-9  # or skip/log instead
intervals.append(value)

When it happens

Trigger: Appending a timedelta/step value that repeats or decreases, e.g. calling monitor.update() twice within the same timer resolution, or computing a duration of 0 due to time.time() truncation, or re-using a stale start timestamp after a checkpoint resume.

Common situations: Very fast steps where perf_counter delta rounds to the same value as before; manually calling the internal timing list; resume-from-checkpoint where the interval list is not reset; mocking time in tests.

Related errors

• Expected lengths ({lengths}) to be greater or equal than sam
• If lengths are passed ({len(self._lengths)}), there needs to
• Device should be CPU, got {device} instead.
• `devices` selected with `CPUAccelerator` should be an int >
• Device should be CUDA, got {device} instead.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/5a77b8d6f8a12c4c. Report an issue: GitHub.