What causes "Making progress out of range" in apache/beam?

Upgrade Beam to a version with improved BlockTracker progress tracking (see TODO BEAM-10842).

apache/beam · error · IOException

Making progress out of range

Error message

Making progress out of range

What it means

ParquetIO's ReadFiles progress tracker (used for split/progress reporting) advances progress by an estimated record size on each call to makeProgress(); if the accumulated progress exceeds totalWork it throws, since the reader claims to have produced more work than the source contains. It is an internal bookkeeping guard against bad size estimates or over-reading.

Solutions

Upgrade Beam to a version with improved BlockTracker progress tracking (see TODO BEAM-10842).
Reduce record size variance in the data or split files into smaller files so estimates stay within totalWork.
If the error aborts the pipeline, disable/work around dynamic splitting for this source.
Report/reproduce with file statistics to help refine approximateRecordSize.

Defensive patterns

Strategy: retry

Validate before calling

// Check approximate size vs totalWork before splitting
if (approximateRecordSize * recordCountEstimate > totalWork) { /* estimate too coarse; split files further */ }

Try / catch

// Reader-level guard
try {
  reader.makeProgress();
} catch (IOException e) {
  if (e.getMessage().contains("Making progress out of range")) { /* restart read or clamp progress */ }
}

Prevention

Keep Beam up to date (progress tracking improvements, BEAM-10842).
Avoid extreme record-size variance or oversized rows in parquet files.
Prefer many smaller files over a few huge ones for better split estimates.

When it happens

Trigger: During Beam splitter/progress reporting when progress + approximateRecordSize exceeds totalWork for the parquet file range.

Common situations: Inaccurate approximateRecordSize estimation on files with very large or variable-size records; very small files where the estimate overruns quickly; running with dynamic work rebalancing.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f748d33ec858b3ab. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java:963

    public static class BlockTracker extends OffsetRangeTracker {
      private long totalWork;
      private long progress;
      private long approximateRecordSize;

      public BlockTracker(OffsetRange range, long totalByteSize, long recordCount) {
        super(range);
        if (recordCount != 0) {
          this.approximateRecordSize = totalByteSize / recordCount;
          // Ensure that totalWork = approximateRecordSize * recordCount
          this.totalWork = approximateRecordSize * recordCount;
          this.progress = 0;
        }
      }

      public void makeProgress() throws IOException {
        progress += approximateRecordSize;
        if (progress > totalWork) {
          throw new IOException("Making progress out of range");
        }
      }

      @Override
      // TODO(BEAM-10842): Refine the BlockTracker to provide better progress.
      public Progress getProgress() {
        return super.getProgress();
      }
    }

    public static class BeamParquetInputFile implements InputFile {
      private final SeekableByteChannel seekableByteChannel;

      public BeamParquetInputFile(SeekableByteChannel seekableByteChannel) {
        this.seekableByteChannel = seekableByteChannel;
      }

      @Override

View on GitHub (pinned to 12126d8942)