{"record":{"id":"3710726c8751b742","repo":"apache/beam","slug":"error-in-computing-the-fractions-consumed-as-recordreader","errorCode":null,"errorMessage":"Error in computing the fractions consumed as RecordReader.getProgress() throws an exception : ","messagePattern":"Error in computing the fractions consumed as RecordReader\\.getProgress\\(\\) throws an exception : ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java","lineNumber":1055,"sourceCode":"          return 0.0;\n        }\n        if (progressValue.get() == 0.0) {\n          return null;\n        }\n        return progressValue.doubleValue();\n      }\n\n      /** Returns RecordReader's progress. */\n      private Double getProgress() throws IOException, InterruptedException {\n        try {\n          float progress = recordReader.getProgress();\n          return (double) progress < 0 || progress > 1 ? 0.0 : progress;\n        } catch (IOException e) {\n          LOG.error(\n              \"Error in computing the fractions consumed as RecordReader.getProgress() throws an \"\n                  + \"exception : \",\n              e);\n          throw new IOException(\n              \"Error in computing the fractions consumed as RecordReader.getProgress() throws an \"\n                  + \"exception : \"\n                  + e.getMessage(),\n              e);\n        }\n      }\n\n      @Override\n      public final long getSplitPointsRemaining() {\n        if (doneReading) {\n          return 0;\n        }\n        /*\n         This source does not currently support dynamic work rebalancing, so remaining parallelism\n         is always 1.\n        */\n        return 1;\n      }","sourceCodeStart":1037,"sourceCodeEnd":1073,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java#L1037-L1073","documentation":"To support dynamic work rebalancing, the reader computes progress via RecordReader.getProgress(). If that call throws an IOException, it is logged and rethrown with this message plus the cause's message, so the runner cannot compute the fraction consumed.","triggerScenarios":"RecordReader.getProgress() throwing — common for InputFormats that do not implement getProgress() (abstract method returning garbage) or whose progress computation hits I/O errors, especially in custom InputFormats or when calling splitAtFraction/dynamic rebalancing on the source.","commonSituations":"Custom RecordReader without a proper getProgress() implementation; underlying stream errors (HDFS read failure) during progress computation; runners like Dataflow triggering dynamic splitting.","solutions":["Implement getProgress() correctly in the custom RecordReader (return bytes-read / total)","Disable dynamic work rebalancing if the InputFormat cannot support progress reporting","Check the chained cause for underlying I/O failures on the input stream"],"exampleFix":"// before\npublic float getProgress() { throw new IOException(\"unsupported\"); }\n// after\npublic float getProgress() throws IOException { return total > 0 ? (float) pos / total : 0.0f; }","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  source.splitAtFraction(fraction);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"fractions consumed\")) {\n    LOG.warn(\"InputFormat does not support progress; disabling dynamic splitting\");\n  }\n}","preventionTips":["Always implement getProgress() in custom RecordReaders","Test dynamic work rebalancing with your InputFormat before production","Ensure the underlying input stream is healthy (checksums, network)"],"tags":["hadoop","progress","fraction-consumed","reader"],"backgroundTag":"method-not-implemented","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}