apache/hadoop · error · IOException

server didn't return all expected map outputs: {remaining.si

Error message

server didn't return all expected map outputs: {remaining.size()} left.

What it means

Fetcher's sanity check after a shuffle HTTP exchange: if no individual map output was reported failed (failedTasks empty) yet the response did not account for every id in 'remaining', the server silently omitted expected map outputs — an IOException that the shuffle scheduler records as a copy failure for later retry.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java:371

          url = getMapOutputURL(host, remaining);
          input = openShuffleUrl(host, remaining, url);
          if (input == null) {
            return;
          }
        }
      }
      
      if(failedTasks != null && failedTasks.length > 0) {
        LOG.warn("copyMapOutput failed for tasks "+Arrays.toString(failedTasks));
        scheduler.hostFailed(host.getHostName());
        for(TaskAttemptID left: failedTasks) {
          scheduler.copyFailed(left, host, true, false);
        }
      }

      // Sanity check
      if (failedTasks == null && !remaining.isEmpty()) {
        throw new IOException("server didn't return all expected map outputs: "
            + remaining.size() + " left.");
      }
      input.close();
      input = null;
    } finally {
      if (input != null) {
        IOUtils.cleanupWithLogger(LOG, input);
        input = null;
      }
      for (TaskAttemptID left : remaining) {
        scheduler.putBackKnownMapOutput(host, left);
      }
    }
  }

  private void setupConnectionsWithRetry(URL url) throws IOException {
    openConnectionWithRetry(url);
    if (stopped) {

View on GitHub (pinned to 2add963021)

Solutions

1. Check the NM shuffle handler log for the host named in the reduce log — outputs may have been cleaned mid-fetch
2. Avoid NM/shuffle-service restarts during shuffle; finish faster with more fetchers (mapreduce.reduce.shuffle.parallelcopies)
3. Keep Hadoop versions aligned across NMs; make custom shuffle handlers reply for every requested map id
4. For a transient occurrence, rely on the scheduler's per-output retry from another host

// Handled by the framework: ShuffleScheduler records the copy failure and
// retries this map output (from the same or another host). No user catch needed;
// only act if the same map output fails past mapreduce.reduce.shuffle.maxfetchfailures.

When it happens

Trigger: The NodeManager shuffle handler's reply lists fewer map-output ids than requested: outputs expired or cleaned mid-fetch (shuffle service restart, NM restart), handler version differences pruning ids, or a custom shuffle plugin that does not echo every requested id.

Common situations: NM or shuffle handler restarts during long shuffles on big jobs; aggressive map-output cleanup before all reducers finish; mixed Hadoop versions on NMs; hand-written shuffle services behind the MapReduce shuffle protocol.

Related errors

• Got invalid response code {rc} from {url}: {responseMessage}
• Incompatible shuffle response version
• read past end of stream reading {mapId}
• {failures} failures downloading {mapId}
• "Mkdirs failed to create " + workDir.toString()

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6812e60b73ef6f0a. Report an issue: GitHub.