What causes "read past end of stream reading {mapId}" in apache/hadoop?

Check the NodeManager log on the source host at the same timestamp for the exception that cut the stream.

apache/hadoop · error · IOException

read past end of stream reading {mapId}

Error message

read past end of stream reading {mapId}

What it means

OnDiskMapOutput copies exactly compressedLength bytes from the checksummed shuffle stream via readWithChecksum; an n < 0 return before the count is satisfied means the peer closed the connection early — a truncated shuffle body relative to the announced length. The scheduler counts it as a fetch failure for that map output and host.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java:110

  static Path getTempPath(Path outPath, int fetcher) {
    return outPath.suffix(String.valueOf(fetcher));
  }

  @Override
  protected void doShuffle(MapHost host, IFileInputStream input,
                      long compressedLength, long decompressedLength,
                      ShuffleClientMetrics metrics,
                      Reporter reporter) throws IOException {
    // Copy data to local-disk
    long bytesLeft = compressedLength;
    try {
      final int BYTES_TO_READ = 64 * 1024;
      byte[] buf = new byte[BYTES_TO_READ];
      while (bytesLeft > 0) {
        int n = input.readWithChecksum(buf, 0,
                                      (int) Math.min(bytesLeft, BYTES_TO_READ));
        if (n < 0) {
          throw new IOException("read past end of stream reading " + 
                                getMapId());
        }
        disk.write(buf, 0, n);
        bytesLeft -= n;
        metrics.inputBytes(n);
        reporter.progress();
      }

      LOG.info("Read " + (compressedLength - bytesLeft) + 
               " bytes from map-output for " + getMapId());

      disk.close();
    } catch (IOException ioe) {
      // Close the streams
      IOUtils.cleanupWithLogger(LOG, disk);

      // Re-throw
      throw ioe;

View on GitHub (pinned to 2add963021)

Solutions

Check the NodeManager log on the source host at the same timestamp for the exception that cut the stream.
Raise mapreduce.reduce.shuffle.read.timeout for jobs with large map outputs.
Verify NM shuffle connection limits and file-descriptor settings are adequate for concurrent reducers.
Restart or drain the failing NM; the framework retries the fetch from another attempt.

Example fix

// before: default read timeout cuts off large spills
conf.setInt("mapreduce.reduce.shuffle.read.timeout", 80000);
// after: allow long streams to finish
conf.setInt("mapreduce.reduce.shuffle.read.timeout", 300000);

Defensive patterns

Strategy: retry

Try / catch

catch (java.io.IOException e) { if (String.valueOf(e.getMessage()).contains("read past end of stream")) { /* truncated transfer: scheduler will re-fetch; investigate NM if one host recurs */ } else { throw e; } }

Prevention

Size mapreduce.reduce.shuffle.read.timeout to your largest map output's transfer time.
Check NM shuffle connection and fd limits when many reducers fetch concurrently.
Drain NodeManagers with recurring mid-stream disconnects.

When it happens

Trigger: NodeManager exception mid-transfer (disk read error on the spill), NM shutdown or restart mid-stream, network reset, server-side timeouts or connection limits killing long transfers of very large map outputs.

Common situations: Very large map outputs streaming slower than mapreduce.reduce.shuffle.read.timeout (default 80000 ms); NM fd/connection caps (shuffle connection limits) dropping transfers; flaky network links or NIC issues between NM and reducer.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f2325696cce4d238. Report an issue: GitHub.