{"record":{"id":"4c9192c6772df9f8","repo":"apache/hadoop","slug":"error-in-shuffle-in-throwingthreadname","errorCode":null,"errorMessage":"error in shuffle in {throwingThreadName}","messagePattern":"error in shuffle in (.+?)","errorType":"exception","errorClass":"ShuffleError","httpStatus":null,"severity":"critical","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java","lineNumber":136,"sourceCode":"          merger, reporter, metrics, this, reduceTask.getShuffleSecret(),\n          localMapFiles);\n      fetchers[0].start();\n    } else {\n      for (int i=0; i < numFetchers; ++i) {\n        fetchers[i] = new Fetcher<K, V>(jobConf, reduceId, scheduler, merger,\n                                       reporter, metrics, this, \n                                       reduceTask.getShuffleSecret());\n        fetchers[i].start();\n      }\n    }\n    \n    // Wait for shuffle to complete successfully\n    while (!scheduler.waitUntilDone(PROGRESS_FREQUENCY)) {\n      reporter.progress();\n      \n      synchronized (this) {\n        if (throwable != null) {\n          throw new ShuffleError(\"error in shuffle in \" + throwingThreadName,\n                                 throwable);\n        }\n      }\n    }\n\n    // Stop the event-fetcher thread\n    eventFetcher.shutDown();\n    \n    // Stop the map-output fetcher threads\n    for (Fetcher<K, V> fetcher : fetchers) {\n      fetcher.shutDown();\n    }\n    \n    // stop the scheduler\n    scheduler.close();\n\n    copyPhase.complete(); // copy is already complete\n    taskStatus.setPhase(TaskStatus.Phase.SORT);","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Shuffle.java#L118-L154","documentation":"Shuffle.run() polls scheduler.waitUntilDone(); any fetcher or event-fetcher thread that hit a fatal error called Shuffle.reportException, which stored the throwable and its thread name. The waiting loop rethrows it wrapped in ShuffleError('error in shuffle in <thread>'), so this message is generic — the real cause is always the attached cause chain.","triggerScenarios":"Repeated fetch failures reaching abortFailureLimit = Math.max(30, totalMaps / 10) for one map output (reported as 'N failures downloading mapId'); fatal per-fetch IOExceptions marked non-retryable; disk full while writing fetched outputs.","commonSituations":"A systemic issue during the copy phase: one or more unhealthy NodeManagers, network partitions, local disk exhaustion on reduce nodes; the reducer log shows this message at the top with the real culprit underneath.","solutions":["Read the ShuffleError cause chain in the reducer log; it identifies the actual failing layer (HTTP errors, truncation, fetch-failure limit, disk).","Fix the underlying host, disk, or network issue named by the cause before changing any configuration.","If the cause is repeated fetch failures, follow up on the specific mapId/host pair in the surrounding 'Failed to fetch' logs.","Rerun the job after remediation; AM retries usually clear transient cluster blips."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable root = e; while (root.getCause() != null) { root = root.getCause(); } LOG.error(\"shuffle failed in thread per: \" + e.getMessage(), root); /* classify: host failure, disk, fetch limit; then resubmit or page */ }","preventionTips":["Never act on the generic message alone; always unwrap the cause chain.","Monitor the 'Failed to fetch' WARN lines that precede it to catch failing hosts early.","Keep abort failure tolerance in mind: max(30, totalMaps/10) fetch failures per map output is fatal."],"tags":["hadoop","mapreduce","shuffle","task-failure","concurrency"],"backgroundTag":"worker-thread-exception","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}