apache/hadoop · critical · ShuffleError

error in shuffle in {throwingThreadName}

Error message

error in shuffle in {throwingThreadName}

What it means

Shuffle.run() polls scheduler.waitUntilDone(); any fetcher or event-fetcher thread that hit a fatal error called Shuffle.reportException, which stored the throwable and its thread name. The waiting loop rethrows it wrapped in ShuffleError('error in shuffle in <thread>'), so this message is generic — the real cause is always the attached cause chain.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java:136

          merger, reporter, metrics, this, reduceTask.getShuffleSecret(),
          localMapFiles);
      fetchers[0].start();
    } else {
      for (int i=0; i < numFetchers; ++i) {
        fetchers[i] = new Fetcher<K, V>(jobConf, reduceId, scheduler, merger,
                                       reporter, metrics, this, 
                                       reduceTask.getShuffleSecret());
        fetchers[i].start();
      }
    }
    
    // Wait for shuffle to complete successfully
    while (!scheduler.waitUntilDone(PROGRESS_FREQUENCY)) {
      reporter.progress();
      
      synchronized (this) {
        if (throwable != null) {
          throw new ShuffleError("error in shuffle in " + throwingThreadName,
                                 throwable);
        }
      }
    }

    // Stop the event-fetcher thread
    eventFetcher.shutDown();
    
    // Stop the map-output fetcher threads
    for (Fetcher<K, V> fetcher : fetchers) {
      fetcher.shutDown();
    }
    
    // stop the scheduler
    scheduler.close();

    copyPhase.complete(); // copy is already complete
    taskStatus.setPhase(TaskStatus.Phase.SORT);

View on GitHub (pinned to 2add963021)

Solutions

1. Read the ShuffleError cause chain in the reducer log; it identifies the actual failing layer (HTTP errors, truncation, fetch-failure limit, disk).
2. Fix the underlying host, disk, or network issue named by the cause before changing any configuration.
3. If the cause is repeated fetch failures, follow up on the specific mapId/host pair in the surrounding 'Failed to fetch' logs.
4. Rerun the job after remediation; AM retries usually clear transient cluster blips.

catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable root = e; while (root.getCause() != null) { root = root.getCause(); } LOG.error("shuffle failed in thread per: " + e.getMessage(), root); /* classify: host failure, disk, fetch limit; then resubmit or page */ }

When it happens

Trigger: Repeated fetch failures reaching abortFailureLimit = Math.max(30, totalMaps / 10) for one map output (reported as 'N failures downloading mapId'); fatal per-fetch IOExceptions marked non-retryable; disk full while writing fetched outputs.

Common situations: A systemic issue during the copy phase: one or more unhealthy NodeManagers, network partitions, local disk exhaustion on reduce nodes; the reducer log shows this message at the top with the real culprit underneath.

Related errors

• Error while doing final merge
• Checksum error reading spill index: " + indexFileName
• One or more threads encountered exception during close. See
• Got invalid response code {rc} from {url}: {responseMessage}
• Incompatible shuffle response version

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4c9192c6772df9f8. Report an issue: GitHub.