{"record":{"id":"8c0ca5be0c598037","repo":"apache/hadoop","slug":"failures-failures-downloading-mapid","errorCode":null,"errorMessage":"{failures} failures downloading {mapId}","messagePattern":"(.+?) failures downloading (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleSchedulerImpl.java","lineNumber":302,"sourceCode":"      failures = x.get();\n    } else {\n      failureCounts.put(mapId, new IntWritable(1));\n    }\n    String hostname = host.getHostName();\n    IntWritable hostFailedNum = hostFailures.get(hostname);\n    // MAPREDUCE-6361: hostname could get cleanup from hostFailures in another\n    // thread with copySucceeded.\n    // In this case, add back hostname to hostFailures to get rid of NPE issue.\n    if (hostFailedNum == null) {\n      hostFailures.put(hostname, new IntWritable(1));\n    }\n    //report failure if already retried maxHostFailures times\n    boolean hostFail = hostFailures.get(hostname).get() >\n        getMaxHostFailures() ? true : false;\n\n    if (failures >= abortFailureLimit) {\n      try {\n        throw new IOException(failures + \" failures downloading \" + mapId);\n      } catch (IOException ie) {\n        reporter.reportException(ie);\n      }\n    }\n\n    checkAndInformMRAppMaster(failures, mapId, readError, connectExcpt,\n        hostFail);\n\n    checkReducerHealth();\n\n    long delay = (long) (INITIAL_PENALTY *\n        Math.pow(PENALTY_GROWTH_RATE, failures));\n    penalize(host, Math.min(delay, maxPenalty));\n\n    failedShuffleCounter.increment(1);\n  }\n\n  /**","sourceCodeStart":284,"sourceCodeEnd":320,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/ShuffleSchedulerImpl.java#L284-L320","documentation":"ShuffleSchedulerImpl.fail() counts fetch failures per map output; once the count reaches abortFailureLimit = Math.max(30, totalMaps / 10), it reports IOException 'N failures downloading <mapId>', which Shuffle.reportException turns into the fatal ShuffleError for the reduce task. The failing host also gets an exponentially growing penalty (INITIAL_PENALTY * PENALTY_GROWTH_RATE^failures) before this point.","triggerScenarios":"One map output is persistently unfetchable: the serving NodeManager is down or unreachable, its spill file was deleted, or every attempt fails with HTTP errors or truncated transfers; the failure count crosses max(30, totalMaps/10).","commonSituations":"A dead or disk-less NodeManager still listed as hosting map outputs; NM local dirs cleaned while reducers lag behind; persistent 4xx/5xx from one host; networks that black-hole one rack.","solutions":["Take the mapId from the message, find the corresponding 'Failed to fetch' log lines, and inspect that NodeManager (process up, disk space, logs).","If the map output no longer exists on that host, ensure the AM re-runs the map (mapreduce.map.maxattempts not exhausted) — restarting or blacklisting the bad NM triggers it.","For very large jobs where totalMaps/10 is a tight budget, treat this as a cluster-health signal: fix or drain failing hosts rather than retrying blindly.","Restart the failing NodeManager to clear stuck shuffle handler state."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable c = e.getCause(); if (c instanceof java.io.IOException && String.valueOf(c.getMessage()).contains(\"failures downloading\")) { /* one map output unfetchable: restart/blacklist the serving NM so the map re-runs, then resubmit */ } else { throw e; } }","preventionTips":["Watch for hosts accumulating 'Failed to fetch' warnings and drain them before the abort limit (max(30, totalMaps/10)) is hit.","Keep NodeManagers that host map outputs alive and their local dirs intact until jobs finish.","For very large jobs, remember the tolerance scales with totalMaps/10 — flaky nodes kill big jobs faster."],"tags":["hadoop","mapreduce","shuffle","fetcher","retry-exhausted","node-failure"],"backgroundTag":"retry-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}