{"record":{"id":"42b304a6f8a71138","repo":"apache/hadoop","slug":"error-while-doing-final-merge","errorCode":null,"errorMessage":"Error while doing final merge ","messagePattern":"Error while doing final merge ","errorType":"exception","errorClass":"ShuffleError","httpStatus":null,"severity":"critical","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java","lineNumber":162,"sourceCode":"    \n    // Stop the map-output fetcher threads\n    for (Fetcher<K, V> fetcher : fetchers) {\n      fetcher.shutDown();\n    }\n    \n    // stop the scheduler\n    scheduler.close();\n\n    copyPhase.complete(); // copy is already complete\n    taskStatus.setPhase(TaskStatus.Phase.SORT);\n    reduceTask.statusUpdate(umbilical);\n\n    // Finish the on-going merges...\n    RawKeyValueIterator kvIter = null;\n    try {\n      kvIter = merger.close();\n    } catch (Throwable e) {\n      throw new ShuffleError(\"Error while doing final merge \", e);\n    }\n\n    // Sanity check\n    synchronized (this) {\n      if (throwable != null) {\n        throw new ShuffleError(\"error in shuffle in \" + throwingThreadName,\n                               throwable);\n      }\n    }\n    \n    return kvIter;\n  }\n\n  @Override\n  public void close(){\n  }\n\n  public synchronized void reportException(Throwable t) {","sourceCodeStart":144,"sourceCodeEnd":180,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java#L144-L180","documentation":"After the copy phase completes, Shuffle.run() calls merger.close() to run the final merge over in-memory segments and on-disk spill files; any Throwable from that merge is wrapped as ShuffleError('Error while doing final merge'). The cause chain holds the real failure, typically local disk exhaustion, corrupt spills, checksum errors, or OOM while merging large segments.","triggerScenarios":"Reduce-side local disk full or failing under yarn.nodemanager.local-dirs; a spill file corrupted or deleted mid-merge; OutOfMemoryError when in-memory segments plus merge buffers exceed the reducer heap.","commonSituations":"Small NM local dirs on busy nodes; reduce tasks with many large map outputs; aggressive mapreduce.reduce.input.buffer.percent settings causing OOM during final merge; disk quotas hit during large sorts.","solutions":["Check free space and health of the reduce node's local dirs (yarn.nodemanager.local-dirs) at failure time.","Unwrap the ShuffleError cause in the log: DiskErrorException means storage, OutOfMemoryError means sizing, ChecksumException means corrupt spills.","If OOM: lower mapreduce.reduce.shuffle.input.buffer.percent / mapreduce.reduce.input.buffer.percent or raise reducer heap.","Retry the job on different nodes after remediation."],"exampleFix":"// before: nearly all shuffle memory retained in-memory into the merge, OOM risk\nconf.setFloat(\"mapreduce.reduce.input.buffer.percent\", 1.0f);\n// after: spill to disk for the reduce phase, merge reads it back\nconf.setFloat(\"mapreduce.reduce.input.buffer.percent\", 0.0f);","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable root = e; while (root.getCause() != null) { root = root.getCause(); } if (root instanceof java.io.DiskErrorException || root.getMessage() != null && root.getMessage().contains(\"No space left\")) { /* free local dirs, then retry */ } else { throw e; } }","preventionTips":["Monitor yarn.nodemanager.local-dirs free space; final merge is disk-heavy.","Size reducer heap against mapreduce.reduce.shuffle.input.buffer.percent to avoid merge-time OOM.","Schedule regular cleanup of stale NM local dirs."],"tags":["hadoop","mapreduce","shuffle","merge","disk","task-failure"],"backgroundTag":"merge-operation-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}