apache/hadoop · critical · ShuffleError

Error while doing final merge

Error message

Error while doing final merge 

What it means

After the copy phase completes, Shuffle.run() calls merger.close() to run the final merge over in-memory segments and on-disk spill files; any Throwable from that merge is wrapped as ShuffleError('Error while doing final merge'). The cause chain holds the real failure, typically local disk exhaustion, corrupt spills, checksum errors, or OOM while merging large segments.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java:162

    
    // Stop the map-output fetcher threads
    for (Fetcher<K, V> fetcher : fetchers) {
      fetcher.shutDown();
    }
    
    // stop the scheduler
    scheduler.close();

    copyPhase.complete(); // copy is already complete
    taskStatus.setPhase(TaskStatus.Phase.SORT);
    reduceTask.statusUpdate(umbilical);

    // Finish the on-going merges...
    RawKeyValueIterator kvIter = null;
    try {
      kvIter = merger.close();
    } catch (Throwable e) {
      throw new ShuffleError("Error while doing final merge ", e);
    }

    // Sanity check
    synchronized (this) {
      if (throwable != null) {
        throw new ShuffleError("error in shuffle in " + throwingThreadName,
                               throwable);
      }
    }
    
    return kvIter;
  }

  @Override
  public void close(){
  }

  public synchronized void reportException(Throwable t) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check free space and health of the reduce node's local dirs (yarn.nodemanager.local-dirs) at failure time.
2. Unwrap the ShuffleError cause in the log: DiskErrorException means storage, OutOfMemoryError means sizing, ChecksumException means corrupt spills.
3. If OOM: lower mapreduce.reduce.shuffle.input.buffer.percent / mapreduce.reduce.input.buffer.percent or raise reducer heap.
4. Retry the job on different nodes after remediation.

Example fix

// before: nearly all shuffle memory retained in-memory into the merge, OOM risk
conf.setFloat("mapreduce.reduce.input.buffer.percent", 1.0f);
// after: spill to disk for the reduce phase, merge reads it back
conf.setFloat("mapreduce.reduce.input.buffer.percent", 0.0f);

catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable root = e; while (root.getCause() != null) { root = root.getCause(); } if (root instanceof java.io.DiskErrorException || root.getMessage() != null && root.getMessage().contains("No space left")) { /* free local dirs, then retry */ } else { throw e; } }

When it happens

Trigger: Reduce-side local disk full or failing under yarn.nodemanager.local-dirs; a spill file corrupted or deleted mid-merge; OutOfMemoryError when in-memory segments plus merge buffers exceed the reducer heap.

Common situations: Small NM local dirs on busy nodes; reduce tasks with many large map outputs; aggressive mapreduce.reduce.input.buffer.percent settings causing OOM during final merge; disk quotas hit during large sorts.

Related errors

• Rec# {recNo}: Failed to skip past key of length: {currentKey
• Rec# {recNo}: Failed to skip past value of length: {currentV
• Invalid configuration: maxSingleShuffleLimit should be less
• error in shuffle in {throwingThreadName}
• Checksum error reading spill index: " + indexFileName

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/42b304a6f8a71138. Report an issue: GitHub.