Lightning-AI/pytorch-lightning · error · RuntimeError

If lengths are passed

Error message

If lengths are passed ({len(self._lengths)}), there needs to be the same number of samples ({len(self._samples)})

What it means

ThroughputMonitor.update() keeps parallel internal lists _samples and _lengths and requires them to stay in lockstep. This RuntimeError fires when lengths was passed on some update() calls but not others, so the list lengths diverge.

Solutions

Pass lengths consistently on every update() call (compute it every iteration, defaulting to the batch size if all sequences are equal length)
Or never pass lengths at all if you do not need per-element accounting
Audit all call sites of update() to ensure the lengths argument is not conditionally omitted

Example fix

# before
lengths = total_tokens if variable else None
throughput.update(batch=b, samples=n, lengths=lengths)

# after
lengths = total_tokens if variable else n  # always a value
throughput.update(batch=b, samples=n, lengths=lengths)

Defensive patterns

Strategy: validation

Validate before calling

def safe_update(mon, **kw):
    has_lengths = len(mon._lengths) > 0
    if has_lengths and kw.get('lengths') is None:
        kw['lengths'] = kw['samples']  # fall back to samples count
    mon.update(**kw)

Prevention

Decide up front whether you track lengths and stick to it for every update call
Wrap update() in one helper so the lengths argument cannot diverge across call sites

When it happens

Trigger: Calling throughput.update(..., lengths=None) on some iterations (e.g. only tracking lengths on certain steps or ranks) after having previously passed a non-None lengths value, making len(self._samples) != len(self._lengths).

Common situations: Conditional tracking like `if step % 10 == 0: update(..., lengths=...)` else `update(...)` without lengths; a code path where lengths is only computed for variable-length batches; or a version upgrade where lengths tracking was added mid-run.

Related errors

AI-assisted analysis of Lightning-AI/pytorch-lightning@9fed5c27d2 (2026-08-28). Data as JSON: /api/errors/dd3773625197f110. Report an issue: GitHub.

Appendix: source

Thrown at src/lightning/fabric/utilities/throughput.py:164

            samples: Total samples seen per device. It should monotonically increase by the batch size with each call.
            lengths: Total length of the samples seen. It should monotonically increase by the lengths of a batch with
                each call.
            flops: Flops elapased per device since last ``update()`` call. You can easily compute this by using
                :func:`measure_flops` and multiplying it by the number of batches that have been processed.
                The value might be different in each device if the batch size is not the same.

        """
        self._time.append(time)
        if samples < batches:
            raise ValueError(f"Expected samples ({samples}) to be greater or equal than batches ({batches})")
        self._batches.append(batches)
        self._samples.append(samples)
        if lengths is not None:
            if lengths < samples:
                raise ValueError(f"Expected lengths ({lengths}) to be greater or equal than samples ({samples})")
            self._lengths.append(lengths)
            if len(self._samples) != len(self._lengths):
                raise RuntimeError(
                    f"If lengths are passed ({len(self._lengths)}), there needs to be the same number of samples"
                    f" ({len(self._samples)})"
                )
        if flops is not None:
            # sum of flops across ranks
            self._flops.append(flops * self.world_size)

    def compute(self) -> _THROUGHPUT_METRICS:
        """Compute throughput metrics."""
        metrics = {
            "time": self._time[-1],
            "batches": self._batches[-1],
            "samples": self._samples[-1],
        }
        if self._lengths:
            metrics["lengths"] = self._lengths[-1]

        add_global_metrics = self.world_size > 1

View on GitHub (pinned to 9fed5c27d2)