apache/hadoop · critical · IOException

{failures} failures downloading {mapId}

Error message

{failures} failures downloading {mapId}

What it means

ShuffleSchedulerImpl.fail() counts fetch failures per map output; once the count reaches abortFailureLimit = Math.max(30, totalMaps / 10), it reports IOException 'N failures downloading <mapId>', which Shuffle.reportException turns into the fatal ShuffleError for the reduce task. The failing host also gets an exponentially growing penalty (INITIAL_PENALTY * PENALTY_GROWTH_RATE^failures) before this point.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleSchedulerImpl.java:302

      failures = x.get();
    } else {
      failureCounts.put(mapId, new IntWritable(1));
    }
    String hostname = host.getHostName();
    IntWritable hostFailedNum = hostFailures.get(hostname);
    // MAPREDUCE-6361: hostname could get cleanup from hostFailures in another
    // thread with copySucceeded.
    // In this case, add back hostname to hostFailures to get rid of NPE issue.
    if (hostFailedNum == null) {
      hostFailures.put(hostname, new IntWritable(1));
    }
    //report failure if already retried maxHostFailures times
    boolean hostFail = hostFailures.get(hostname).get() >
        getMaxHostFailures() ? true : false;

    if (failures >= abortFailureLimit) {
      try {
        throw new IOException(failures + " failures downloading " + mapId);
      } catch (IOException ie) {
        reporter.reportException(ie);
      }
    }

    checkAndInformMRAppMaster(failures, mapId, readError, connectExcpt,
        hostFail);

    checkReducerHealth();

    long delay = (long) (INITIAL_PENALTY *
        Math.pow(PENALTY_GROWTH_RATE, failures));
    penalize(host, Math.min(delay, maxPenalty));

    failedShuffleCounter.increment(1);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Take the mapId from the message, find the corresponding 'Failed to fetch' log lines, and inspect that NodeManager (process up, disk space, logs).
2. If the map output no longer exists on that host, ensure the AM re-runs the map (mapreduce.map.maxattempts not exhausted) — restarting or blacklisting the bad NM triggers it.
3. For very large jobs where totalMaps/10 is a tight budget, treat this as a cluster-health signal: fix or drain failing hosts rather than retrying blindly.
4. Restart the failing NodeManager to clear stuck shuffle handler state.

catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable c = e.getCause(); if (c instanceof java.io.IOException && String.valueOf(c.getMessage()).contains("failures downloading")) { /* one map output unfetchable: restart/blacklist the serving NM so the map re-runs, then resubmit */ } else { throw e; } }

When it happens

Trigger: One map output is persistently unfetchable: the serving NodeManager is down or unreachable, its spill file was deleted, or every attempt fails with HTTP errors or truncated transfers; the failure count crosses max(30, totalMaps/10).

Common situations: A dead or disk-less NodeManager still listed as hosting map outputs; NM local dirs cleaned while reducers lag behind; persistent 4xx/5xx from one host; networks that black-hole one rack.

Related errors

• Got invalid response code {rc} from {url}: {responseMessage}
• read past end of stream reading {mapId}
• Checksum error reading spill index: " + indexFileName
• server didn't return all expected map outputs: {remaining.si
• Incompatible shuffle response version

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8c0ca5be0c598037. Report an issue: GitHub.