Lightning-AI/pytorch-lightning · error · RuntimeError

If lengths are passed ({len(self._lengths)}), there needs to

Error message

If lengths are passed ({len(self._lengths)}), there needs to be the same number of samples ({len(self._samples)})

What it means

ThroughputMonitor.update() keeps parallel internal lists _samples and _lengths and requires them to stay in lockstep. This RuntimeError fires when lengths was passed on some update() calls but not others, so the list lengths diverge.

Source

Thrown at src/lightning/fabric/utilities/throughput.py:164

            samples: Total samples seen per device. It should monotonically increase by the batch size with each call.
            lengths: Total length of the samples seen. It should monotonically increase by the lengths of a batch with
                each call.
            flops: Flops elapased per device since last ``update()`` call. You can easily compute this by using
                :func:`measure_flops` and multiplying it by the number of batches that have been processed.
                The value might be different in each device if the batch size is not the same.

        """
        self._time.append(time)
        if samples < batches:
            raise ValueError(f"Expected samples ({samples}) to be greater or equal than batches ({batches})")
        self._batches.append(batches)
        self._samples.append(samples)
        if lengths is not None:
            if lengths < samples:
                raise ValueError(f"Expected lengths ({lengths}) to be greater or equal than samples ({samples})")
            self._lengths.append(lengths)
            if len(self._samples) != len(self._lengths):
                raise RuntimeError(
                    f"If lengths are passed ({len(self._lengths)}), there needs to be the same number of samples"
                    f" ({len(self._samples)})"
                )
        if flops is not None:
            # sum of flops across ranks
            self._flops.append(flops * self.world_size)

    def compute(self) -> _THROUGHPUT_METRICS:
        """Compute throughput metrics."""
        metrics = {
            "time": self._time[-1],
            "batches": self._batches[-1],
            "samples": self._samples[-1],
        }
        if self._lengths:
            metrics["lengths"] = self._lengths[-1]

        add_global_metrics = self.world_size > 1

View on GitHub (pinned to 9fed5c27d2)

Solutions

1. Pass lengths consistently on every update() call (compute it every iteration, defaulting to the batch size if all sequences are equal length)
2. Or never pass lengths at all if you do not need per-element accounting
3. Audit all call sites of update() to ensure the lengths argument is not conditionally omitted

Example fix

# before
lengths = total_tokens if variable else None
throughput.update(batch=b, samples=n, lengths=lengths)

# after
lengths = total_tokens if variable else n  # always a value
throughput.update(batch=b, samples=n, lengths=lengths)

def safe_update(mon, **kw):
    has_lengths = len(mon._lengths) > 0
    if has_lengths and kw.get('lengths') is None:
        kw['lengths'] = kw['samples']  # fall back to samples count
    mon.update(**kw)

When it happens

Trigger: Calling throughput.update(..., lengths=None) on some iterations (e.g. only tracking lengths on certain steps or ranks) after having previously passed a non-None lengths value, making len(self._samples) != len(self._lengths).

Common situations: Conditional tracking like `if step % 10 == 0: update(..., lengths=...)` else `update(...)` without lengths; a code path where lengths is only computed for variable-length batches; or a version upgrade where lengths tracking was added mid-run.

Related errors

• Expected lengths ({lengths}) to be greater or equal than sam
• Expected the value to increase, last: {last}, current: {x}
• Device should be CPU, got {device} instead.
• `devices` selected with `CPUAccelerator` should be an int >
• Device should be CUDA, got {device} instead.

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/dd3773625197f110. Report an issue: GitHub.