{"record":{"id":"f2325696cce4d238","repo":"apache/hadoop","slug":"read-past-end-of-stream-reading-mapid","errorCode":null,"errorMessage":"read past end of stream reading {mapId}","messagePattern":"read past end of stream reading (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java","lineNumber":110,"sourceCode":"  static Path getTempPath(Path outPath, int fetcher) {\n    return outPath.suffix(String.valueOf(fetcher));\n  }\n\n  @Override\n  protected void doShuffle(MapHost host, IFileInputStream input,\n                      long compressedLength, long decompressedLength,\n                      ShuffleClientMetrics metrics,\n                      Reporter reporter) throws IOException {\n    // Copy data to local-disk\n    long bytesLeft = compressedLength;\n    try {\n      final int BYTES_TO_READ = 64 * 1024;\n      byte[] buf = new byte[BYTES_TO_READ];\n      while (bytesLeft > 0) {\n        int n = input.readWithChecksum(buf, 0,\n                                      (int) Math.min(bytesLeft, BYTES_TO_READ));\n        if (n < 0) {\n          throw new IOException(\"read past end of stream reading \" + \n                                getMapId());\n        }\n        disk.write(buf, 0, n);\n        bytesLeft -= n;\n        metrics.inputBytes(n);\n        reporter.progress();\n      }\n\n      LOG.info(\"Read \" + (compressedLength - bytesLeft) + \n               \" bytes from map-output for \" + getMapId());\n\n      disk.close();\n    } catch (IOException ioe) {\n      // Close the streams\n      IOUtils.cleanupWithLogger(LOG, disk);\n\n      // Re-throw\n      throw ioe;","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java#L92-L128","documentation":"OnDiskMapOutput copies exactly compressedLength bytes from the checksummed shuffle stream via readWithChecksum; an n < 0 return before the count is satisfied means the peer closed the connection early — a truncated shuffle body relative to the announced length. The scheduler counts it as a fetch failure for that map output and host.","triggerScenarios":"NodeManager exception mid-transfer (disk read error on the spill), NM shutdown or restart mid-stream, network reset, server-side timeouts or connection limits killing long transfers of very large map outputs.","commonSituations":"Very large map outputs streaming slower than mapreduce.reduce.shuffle.read.timeout (default 80000 ms); NM fd/connection caps (shuffle connection limits) dropping transfers; flaky network links or NIC issues between NM and reducer.","solutions":["Check the NodeManager log on the source host at the same timestamp for the exception that cut the stream.","Raise mapreduce.reduce.shuffle.read.timeout for jobs with large map outputs.","Verify NM shuffle connection limits and file-descriptor settings are adequate for concurrent reducers.","Restart or drain the failing NM; the framework retries the fetch from another attempt."],"exampleFix":"// before: default read timeout cuts off large spills\nconf.setInt(\"mapreduce.reduce.shuffle.read.timeout\", 80000);\n// after: allow long streams to finish\nconf.setInt(\"mapreduce.reduce.shuffle.read.timeout\", 300000);","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains(\"read past end of stream\")) { /* truncated transfer: scheduler will re-fetch; investigate NM if one host recurs */ } else { throw e; } }","preventionTips":["Size mapreduce.reduce.shuffle.read.timeout to your largest map output's transfer time.","Check NM shuffle connection and fd limits when many reducers fetch concurrently.","Drain NodeManagers with recurring mid-stream disconnects."],"tags":["hadoop","mapreduce","shuffle","network","truncated-response","fetcher"],"backgroundTag":"truncated-response-body","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}