{"record":{"id":"620633feb8cbd8fe","repo":"apache/hadoop","slug":"incomplete-map-output-received-for-mapid-from-h","errorCode":null,"errorMessage":"Incomplete map output received for {mapId} from {hostName} ({bytesLeft} bytes missing of {compressedLength})","messagePattern":"Incomplete map output received for (.+?) from (.+?) \\((.+?) bytes missing of (.+?)\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java","lineNumber":133,"sourceCode":"        metrics.inputBytes(n);\n        reporter.progress();\n      }\n\n      LOG.info(\"Read \" + (compressedLength - bytesLeft) + \n               \" bytes from map-output for \" + getMapId());\n\n      disk.close();\n    } catch (IOException ioe) {\n      // Close the streams\n      IOUtils.cleanupWithLogger(LOG, disk);\n\n      // Re-throw\n      throw ioe;\n    }\n\n    // Sanity check\n    if (bytesLeft != 0) {\n      throw new IOException(\"Incomplete map output received for \" +\n                            getMapId() + \" from \" +\n                            host.getHostName() + \" (\" + \n                            bytesLeft + \" bytes missing of \" + \n                            compressedLength + \")\");\n    }\n    this.compressedSize = compressedLength;\n  }\n\n  @Override\n  public void commit() throws IOException {\n    fs.rename(tmpOutputPath, outputPath);\n    CompressAwarePath compressAwarePath = new CompressAwarePath(outputPath,\n        getSize(), this.compressedSize);\n    getMerger().closeOnDiskFile(compressAwarePath);\n  }\n  \n  @Override\n  public void abort() {","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java#L115-L151","documentation":"OnDiskMapOutput's copy loop exits when bytesLeft <= 0, and this post-loop sanity check requires exactly 0 remaining. Reaching it with bytesLeft != 0 means readWithChecksum consumed more than requested — effectively a defensive guard against length-accounting bugs, reporting the same class of truncated/mismatched transfer as the read-past-end check.","triggerScenarios":"An IFileInputStream/checksum stream returning more bytes than requested for a single read; in practice nearly unreachable — the loop's EOF case (n < 0) fails earlier with 'read past end of stream'.","commonSituations":"Seen together with other truncation symptoms during shuffle stream corruption investigations; essentially a defensive assertion rather than a routine failure mode.","solutions":["Treat it like a truncated transfer: gather NodeManager logs for the source host and the announced compressedLength.","Retry the job; if reproducible with a specific codec or Hadoop version, capture the fetch details and report upstream.","Verify shuffle stream integrity end-to-end (no proxies, no transparent decompression on the path)."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains(\"Incomplete map output received\")) { /* length mismatch on transfer: collect NM logs; scheduler retries */ } else { throw e; } }","preventionTips":["Treat as a truncation/corruption signal and gather the announced compressedLength from logs.","Keep the shuffle path free of transparent proxies that alter bodies or headers."],"tags":["hadoop","mapreduce","shuffle","truncated-response","validation"],"backgroundTag":"truncated-response-body","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}