{"record":{"id":"f748d33ec858b3ab","repo":"apache/beam","slug":"making-progress-out-of-range","errorCode":null,"errorMessage":"Making progress out of range","messagePattern":"Making progress out of range","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java","lineNumber":963,"sourceCode":"    public static class BlockTracker extends OffsetRangeTracker {\n      private long totalWork;\n      private long progress;\n      private long approximateRecordSize;\n\n      public BlockTracker(OffsetRange range, long totalByteSize, long recordCount) {\n        super(range);\n        if (recordCount != 0) {\n          this.approximateRecordSize = totalByteSize / recordCount;\n          // Ensure that totalWork = approximateRecordSize * recordCount\n          this.totalWork = approximateRecordSize * recordCount;\n          this.progress = 0;\n        }\n      }\n\n      public void makeProgress() throws IOException {\n        progress += approximateRecordSize;\n        if (progress > totalWork) {\n          throw new IOException(\"Making progress out of range\");\n        }\n      }\n\n      @Override\n      // TODO(BEAM-10842): Refine the BlockTracker to provide better progress.\n      public Progress getProgress() {\n        return super.getProgress();\n      }\n    }\n\n    public static class BeamParquetInputFile implements InputFile {\n      private final SeekableByteChannel seekableByteChannel;\n\n      public BeamParquetInputFile(SeekableByteChannel seekableByteChannel) {\n        this.seekableByteChannel = seekableByteChannel;\n      }\n\n      @Override","sourceCodeStart":945,"sourceCodeEnd":981,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java#L945-L981","documentation":"ParquetIO's ReadFiles progress tracker (used for split/progress reporting) advances progress by an estimated record size on each call to makeProgress(); if the accumulated progress exceeds totalWork it throws, since the reader claims to have produced more work than the source contains. It is an internal bookkeeping guard against bad size estimates or over-reading.","triggerScenarios":"During Beam splitter/progress reporting when progress + approximateRecordSize exceeds totalWork for the parquet file range.","commonSituations":"Inaccurate approximateRecordSize estimation on files with very large or variable-size records; very small files where the estimate overruns quickly; running with dynamic work rebalancing.","solutions":["Upgrade Beam to a version with improved BlockTracker progress tracking (see TODO BEAM-10842).","Reduce record size variance in the data or split files into smaller files so estimates stay within totalWork.","If the error aborts the pipeline, disable/work around dynamic splitting for this source.","Report/reproduce with file statistics to help refine approximateRecordSize."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Check approximate size vs totalWork before splitting\nif (approximateRecordSize * recordCountEstimate > totalWork) { /* estimate too coarse; split files further */ }","typeGuard":null,"tryCatchPattern":"// Reader-level guard\ntry {\n  reader.makeProgress();\n} catch (IOException e) {\n  if (e.getMessage().contains(\"Making progress out of range\")) { /* restart read or clamp progress */ }\n}","preventionTips":["Keep Beam up to date (progress tracking improvements, BEAM-10842).","Avoid extreme record-size variance or oversized rows in parquet files.","Prefer many smaller files over a few huge ones for better split estimates."],"tags":["parquet","progress-tracking","apache-beam","internal"],"backgroundTag":"internal-invariant-violation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}