{"record":{"id":"4770bda1cf4ceeb6","repo":"apache/hadoop","slug":"unexpected-extra-bytes-from-input-stream-for-mapi","errorCode":null,"errorMessage":"Unexpected extra bytes from input stream for {mapId}","messagePattern":"Unexpected extra bytes from input stream for (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java","lineNumber":105,"sourceCode":"      input = codec.createInputStream(input, decompressor);\n    }\n  \n    try {\n      IOUtils.readFully(input, memory, 0, memory.length);\n      metrics.inputBytes(memory.length);\n      reporter.progress();\n      LOG.info(\"Read \" + memory.length + \" bytes from map-output for \" +\n                getMapId());\n\n      /**\n       * We've gotten the amount of data we were expecting. Verify the\n       * decompressor has nothing more to offer. This action also forces the\n       * decompressor to read any trailing bytes that weren't critical\n       * for decompression, which is necessary to keep the stream\n       * in sync.\n       */\n      if (input.read() >= 0 ) {\n        throw new IOException(\"Unexpected extra bytes from input stream for \" +\n                               getMapId());\n      }\n    } finally {\n      CodecPool.returnDecompressor(decompressor);\n    }\n  }\n\n  @Override\n  public void commit() throws IOException {\n    getMerger().closeInMemoryFile(this);\n  }\n  \n  @Override\n  public void abort() {\n    getMerger().unreserve(memory.length);\n  }\n\n  @Override","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/InMemoryMapOutput.java#L87-L123","documentation":"InMemoryMapOutput.load() reads exactly decompressedLength bytes into memory, then calls input.read() once to force the decompressor to drain trailing bytes and keep the stream in sync. A non-negative result means the decompressed stream contains more data than the shuffle header announced, i.e. corrupt length or payload. The scheduler treats it as a fetch failure and re-fetches the map output.","triggerScenarios":"Map output compressed with a different codec than the one used for decompression; corrupted bytes in transit; a NodeManager serving a partially overwritten or truncated spill file so the announced length no longer matches the data.","commonSituations":"Custom job setup code overriding mapreduce.map.output.compress / codec inconsistently across tasks of the same job; NM disk corruption; network gear mangling payloads; rare codec bugs on specific Hadoop versions.","solutions":["Confirm mapreduce.map.output.compress and mapreduce.map.output.compress.codec are identical for all maps of the job (set only on the job conf, never per-task).","Check the source NodeManager's disk and logs for read errors on the spill file.","Retry the job: transient corruption usually clears on a re-fetch or re-run of the map.","If reproducible for one map output, capture the mapId/host pair and inspect the spill file on that NM."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains(\"Unexpected extra bytes from input stream\")) { /* corrupt map output: let the scheduler re-fetch; investigate source NM if it repeats for one mapId */ } else { throw e; } }","preventionTips":["Set compression options (mapreduce.map.output.compress, codec) only on the job conf, never per-task.","Monitor NodeManager disk health; corrupt spills are a leading source of length mismatches.","Rely on the scheduler's built-in re-fetch before escalating to task failure."],"tags":["hadoop","mapreduce","shuffle","data-corruption","compression"],"backgroundTag":"data-corruption-detected","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}