apache/beam · error · IOException

Error in computing the fractions consumed as…

Error message

Error in computing the fractions consumed as RecordReader.getProgress() throws an exception : 

What it means

To support dynamic work rebalancing, the reader computes progress via RecordReader.getProgress(). If that call throws an IOException, it is logged and rethrown with this message plus the cause's message, so the runner cannot compute the fraction consumed.

Solutions

1. Implement getProgress() correctly in the custom RecordReader (return bytes-read / total)
2. Disable dynamic work rebalancing if the InputFormat cannot support progress reporting
3. Check the chained cause for underlying I/O failures on the input stream

Example fix

// before
public float getProgress() { throw new IOException("unsupported"); }
// after
public float getProgress() throws IOException { return total > 0 ? (float) pos / total : 0.0f; }

try {
  source.splitAtFraction(fraction);
} catch (IOException e) {
  if (e.getMessage().contains("fractions consumed")) {
    LOG.warn("InputFormat does not support progress; disabling dynamic splitting");
  }
}

When it happens

Trigger: RecordReader.getProgress() throwing — common for InputFormats that do not implement getProgress() (abstract method returning garbage) or whose progress computation hits I/O errors, especially in custom InputFormats or when calling splitAtFraction/dynamic rebalancing on the source.

Common situations: Custom RecordReader without a proper getProgress() implementation; underlying stream errors (HDFS read failure) during progress computation; runners like Dataflow triggering dynamic splitting.

Related errors

• Cannot create reader as source is not split yet.
• Could not read because the thread got interrupted while…
• Unable to read data:
• Cannot provide because does not implement the interface
• Cannot read format

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3710726c8751b742. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java:1055

          return 0.0;
        }
        if (progressValue.get() == 0.0) {
          return null;
        }
        return progressValue.doubleValue();
      }

      /** Returns RecordReader's progress. */
      private Double getProgress() throws IOException, InterruptedException {
        try {
          float progress = recordReader.getProgress();
          return (double) progress < 0 || progress > 1 ? 0.0 : progress;
        } catch (IOException e) {
          LOG.error(
              "Error in computing the fractions consumed as RecordReader.getProgress() throws an "
                  + "exception : ",
              e);
          throw new IOException(
              "Error in computing the fractions consumed as RecordReader.getProgress() throws an "
                  + "exception : "
                  + e.getMessage(),
              e);
        }
      }

      @Override
      public final long getSplitPointsRemaining() {
        if (doneReading) {
          return 0;
        }
        /*
         This source does not currently support dynamic work rebalancing, so remaining parallelism
         is always 1.
        */
        return 1;
      }

View on GitHub (pinned to 12126d8942)