{"record":{"id":"7f452e7269f2a259","repo":"apache/hadoop","slug":"too-many-failures-downloading-events","errorCode":null,"errorMessage":"too many failures downloading events","messagePattern":"too many failures downloading events","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/EventFetcher.java","lineNumber":82,"sourceCode":"      while (!stopped && !Thread.currentThread().isInterrupted()) {\n        try {\n          int numNewMaps = getMapCompletionEvents();\n          failures = 0;\n          if (numNewMaps > 0) {\n            LOG.info(reduce + \": \" + \"Got \" + numNewMaps + \" new map-outputs\");\n          }\n          LOG.debug(\"GetMapEventsThread about to sleep for \" + SLEEP_TIME);\n          if (!Thread.currentThread().isInterrupted()) {\n            Thread.sleep(SLEEP_TIME);\n          }\n        } catch (InterruptedException e) {\n          LOG.info(\"EventFetcher is interrupted.. Returning\");\n          return;\n        } catch (IOException ie) {\n          LOG.info(\"Exception in getting events\", ie);\n          // check to see whether to abort\n          if (++failures >= MAX_RETRIES) {\n            throw new IOException(\"too many failures downloading events\", ie);\n          }\n          // sleep for a bit\n          if (!Thread.currentThread().isInterrupted()) {\n            Thread.sleep(RETRY_PERIOD);\n          }\n        }\n      }\n    } catch (InterruptedException e) {\n      return;\n    } catch (Throwable t) {\n      exceptionReporter.reportException(t);\n      return;\n    }\n  }\n\n  public void shutDown() {\n    this.stopped = true;\n    interrupt();","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/EventFetcher.java#L64-L100","documentation":"EventFetcher polls the ApplicationMaster every second for map-completion events over the task umbilical RPC. IOExceptions are logged and retried after 5 seconds; after MAX_RETRIES (10) consecutive failures it throws IOException('too many failures downloading events'), which is reported via exceptionReporter and fails the reduce task.","triggerScenarios":"umbilical.getMapCompletionEvents failing continuously for ~50 seconds: AM restart mid-shuffle, RM/AM overload or GC pauses, network partition between the reduce's node and the AM, or the job already killed and its AM gone.","commonSituations":"AM failover during the shuffle phase of large jobs; heavily loaded RMs; firewall/proxy ACLs blocking NM-to-AM traffic for specific nodes; jobs killed by queue limits while reducers run.","solutions":["Check RM UI / AM logs for restarts and the underlying RPC errors — an AM restart during shuffle is the usual cause and task retry recovers","Verify the reduce host can reach the AM (hadoop.registry / proxy ACLs, firewall rules)","If the AM is chronically overloaded, reduce concurrent shuffle load or scale RM/AM resources","Catch the failure in the driver and resubmit if the job died from it"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  boolean ok = job.waitForCompletion(true);\n  if (!ok) { /* framework retries already exhausted */ }\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"too many failures downloading events\")) {\n    // AM-side event fetch failed past its retry budget; resubmit once\n    resubmitJob(job);\n  } else { throw e; }\n}","preventionTips":["Watch AM restart counts during long shuffle phases","Keep NM-to-AM network paths and proxy ACLs open","Reduce AM load during heavy shuffle windows (stagger large jobs)"],"tags":["mapreduce","shuffle","rpc","resourcemanager","retry","app-master"],"backgroundTag":"retry-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}