apache/hadoop · error · IOException

too many failures downloading events

Error message

too many failures downloading events

What it means

EventFetcher polls the ApplicationMaster every second for map-completion events over the task umbilical RPC. IOExceptions are logged and retried after 5 seconds; after MAX_RETRIES (10) consecutive failures it throws IOException('too many failures downloading events'), which is reported via exceptionReporter and fails the reduce task.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/EventFetcher.java:82

      while (!stopped && !Thread.currentThread().isInterrupted()) {
        try {
          int numNewMaps = getMapCompletionEvents();
          failures = 0;
          if (numNewMaps > 0) {
            LOG.info(reduce + ": " + "Got " + numNewMaps + " new map-outputs");
          }
          LOG.debug("GetMapEventsThread about to sleep for " + SLEEP_TIME);
          if (!Thread.currentThread().isInterrupted()) {
            Thread.sleep(SLEEP_TIME);
          }
        } catch (InterruptedException e) {
          LOG.info("EventFetcher is interrupted.. Returning");
          return;
        } catch (IOException ie) {
          LOG.info("Exception in getting events", ie);
          // check to see whether to abort
          if (++failures >= MAX_RETRIES) {
            throw new IOException("too many failures downloading events", ie);
          }
          // sleep for a bit
          if (!Thread.currentThread().isInterrupted()) {
            Thread.sleep(RETRY_PERIOD);
          }
        }
      }
    } catch (InterruptedException e) {
      return;
    } catch (Throwable t) {
      exceptionReporter.reportException(t);
      return;
    }
  }

  public void shutDown() {
    this.stopped = true;
    interrupt();

View on GitHub (pinned to 2add963021)

Solutions

1. Check RM UI / AM logs for restarts and the underlying RPC errors — an AM restart during shuffle is the usual cause and task retry recovers
2. Verify the reduce host can reach the AM (hadoop.registry / proxy ACLs, firewall rules)
3. If the AM is chronically overloaded, reduce concurrent shuffle load or scale RM/AM resources
4. Catch the failure in the driver and resubmit if the job died from it

try {
  boolean ok = job.waitForCompletion(true);
  if (!ok) { /* framework retries already exhausted */ }
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("too many failures downloading events")) {
    // AM-side event fetch failed past its retry budget; resubmit once
    resubmitJob(job);
  } else { throw e; }
}

When it happens

Trigger: umbilical.getMapCompletionEvents failing continuously for ~50 seconds: AM restart mid-shuffle, RM/AM overload or GC pauses, network partition between the reduce's node and the AM, or the job already killed and its AM gone.

Common situations: AM failover during the shuffle phase of large jobs; heavily loaded RMs; firewall/proxy ACLs blocking NM-to-AM traffic for specific nodes; jobs killed by queue limits while reducers run.

Related errors

• Interrupted waiting for the proxy
• Client already attempted negotiation
• Namenode is in startup mode
• Incompatible with LocalRunner
• Unknown Job {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7f452e7269f2a259. Report an issue: GitHub.