apache/hadoop · error · IOException

Incomplete map output received for {mapId} from {hostName} (

Error message

Incomplete map output received for {mapId} from {hostName} ({bytesLeft} bytes missing of {compressedLength})

What it means

OnDiskMapOutput's copy loop exits when bytesLeft <= 0, and this post-loop sanity check requires exactly 0 remaining. Reaching it with bytesLeft != 0 means readWithChecksum consumed more than requested — effectively a defensive guard against length-accounting bugs, reporting the same class of truncated/mismatched transfer as the read-past-end check.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java:133

        metrics.inputBytes(n);
        reporter.progress();
      }

      LOG.info("Read " + (compressedLength - bytesLeft) + 
               " bytes from map-output for " + getMapId());

      disk.close();
    } catch (IOException ioe) {
      // Close the streams
      IOUtils.cleanupWithLogger(LOG, disk);

      // Re-throw
      throw ioe;
    }

    // Sanity check
    if (bytesLeft != 0) {
      throw new IOException("Incomplete map output received for " +
                            getMapId() + " from " +
                            host.getHostName() + " (" + 
                            bytesLeft + " bytes missing of " + 
                            compressedLength + ")");
    }
    this.compressedSize = compressedLength;
  }

  @Override
  public void commit() throws IOException {
    fs.rename(tmpOutputPath, outputPath);
    CompressAwarePath compressAwarePath = new CompressAwarePath(outputPath,
        getSize(), this.compressedSize);
    getMerger().closeOnDiskFile(compressAwarePath);
  }
  
  @Override
  public void abort() {

View on GitHub (pinned to 2add963021)

Solutions

1. Treat it like a truncated transfer: gather NodeManager logs for the source host and the announced compressedLength.
2. Retry the job; if reproducible with a specific codec or Hadoop version, capture the fetch details and report upstream.
3. Verify shuffle stream integrity end-to-end (no proxies, no transparent decompression on the path).

catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains("Incomplete map output received")) { /* length mismatch on transfer: collect NM logs; scheduler retries */ } else { throw e; } }

When it happens

Trigger: An IFileInputStream/checksum stream returning more bytes than requested for a single read; in practice nearly unreachable — the loop's EOF case (n < 0) fails earlier with 'read past end of stream'.

Common situations: Seen together with other truncation symptoms during shuffle stream corruption investigations; essentially a defensive assertion rather than a routine failure mode.

Related errors

• Negative key-length not allowed: {keyLength} for {key}
• Negative value-length not allowed: {valueLength} for {value}
• Invalid value for mapreduce.reduce.shuffle.input.buffer.perc
• Invalid value for mapreduce.reduce.shuffle.memory.limit.perc
• Invalid configuration: maxSingleShuffleLimit should be less

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/620633feb8cbd8fe. Report an issue: GitHub.