{"record":{"id":"6812e60b73ef6f0a","repo":"apache/hadoop","slug":"server-didn-t-return-all-expected-map-outputs-re","errorCode":null,"errorMessage":"server didn't return all expected map outputs: {remaining.size()} left.","messagePattern":"server didn't return all expected map outputs: (.+?) left\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java","lineNumber":371,"sourceCode":"          url = getMapOutputURL(host, remaining);\n          input = openShuffleUrl(host, remaining, url);\n          if (input == null) {\n            return;\n          }\n        }\n      }\n      \n      if(failedTasks != null && failedTasks.length > 0) {\n        LOG.warn(\"copyMapOutput failed for tasks \"+Arrays.toString(failedTasks));\n        scheduler.hostFailed(host.getHostName());\n        for(TaskAttemptID left: failedTasks) {\n          scheduler.copyFailed(left, host, true, false);\n        }\n      }\n\n      // Sanity check\n      if (failedTasks == null && !remaining.isEmpty()) {\n        throw new IOException(\"server didn't return all expected map outputs: \"\n            + remaining.size() + \" left.\");\n      }\n      input.close();\n      input = null;\n    } finally {\n      if (input != null) {\n        IOUtils.cleanupWithLogger(LOG, input);\n        input = null;\n      }\n      for (TaskAttemptID left : remaining) {\n        scheduler.putBackKnownMapOutput(host, left);\n      }\n    }\n  }\n\n  private void setupConnectionsWithRetry(URL url) throws IOException {\n    openConnectionWithRetry(url);\n    if (stopped) {","sourceCodeStart":353,"sourceCodeEnd":389,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java#L353-L389","documentation":"Fetcher's sanity check after a shuffle HTTP exchange: if no individual map output was reported failed (failedTasks empty) yet the response did not account for every id in 'remaining', the server silently omitted expected map outputs — an IOException that the shuffle scheduler records as a copy failure for later retry.","triggerScenarios":"The NodeManager shuffle handler's reply lists fewer map-output ids than requested: outputs expired or cleaned mid-fetch (shuffle service restart, NM restart), handler version differences pruning ids, or a custom shuffle plugin that does not echo every requested id.","commonSituations":"NM or shuffle handler restarts during long shuffles on big jobs; aggressive map-output cleanup before all reducers finish; mixed Hadoop versions on NMs; hand-written shuffle services behind the MapReduce shuffle protocol.","solutions":["Check the NM shuffle handler log for the host named in the reduce log — outputs may have been cleaned mid-fetch","Avoid NM/shuffle-service restarts during shuffle; finish faster with more fetchers (mapreduce.reduce.shuffle.parallelcopies)","Keep Hadoop versions aligned across NMs; make custom shuffle handlers reply for every requested map id","For a transient occurrence, rely on the scheduler's per-output retry from another host"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Handled by the framework: ShuffleScheduler records the copy failure and\n// retries this map output (from the same or another host). No user catch needed;\n// only act if the same map output fails past mapreduce.reduce.shuffle.maxfetchfailures.","preventionTips":["Do not restart NMs or the aux shuffle service mid-shuffle on big jobs","Tune mapreduce.reduce.shuffle.parallelcopies so shuffles finish sooner","Verify custom shuffle handlers respond for every requested map id; keep cluster versions aligned"],"tags":["mapreduce","shuffle","fetcher","node-manager","protocol"],"backgroundTag":"incomplete-server-response","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}