apache/hadoop · error · IOException

Got invalid response code

Error message

Got invalid response code {rc} from {url}: {responseMessage}

What it means

Thrown by the reduce-side Fetcher when an HTTP shuffle request to a NodeManager's shuffle handler returns a status other than 200 OK (and other than 429, which is converted to TryAgainLaterException honoring the Retry-After header). This is a per-map-output fetch failure: ShuffleSchedulerImpl penalizes the host with an exponential backoff and re-queues the output, so the reducer only dies if the failure repeats past abortFailureLimit.

Solutions

1. Correlate with the NodeManager log on the host named in the URL at the same timestamp; the server-side stack trace explains the non-200 code.
2. If 404/400: the map attempt is stale on that node; the framework re-fetches another attempt, so verify the AM re-ran the map and that failure tolerance covers the window.
3. If 401/403: check job-token propagation and that client and cluster share the same hadoop.security.authentication setup.
4. If a proxy sits in the shuffle path, remove it or make it pass the NodeManager shuffle aux-service port through without rewriting the response.

Example fix

// before: large map outputs time out server-side and yield 5xx; small read timeout
conf.setInt("mapreduce.reduce.shuffle.read.timeout", 80000);
// after: give slow NodeManagers time to stream big spills
conf.setInt("mapreduce.reduce.shuffle.read.timeout", 240000);
conf.setInt("mapreduce.reduce.shuffle.connect.timeout", 180000);

// Pre-flight: verify the NodeManager shuffle endpoint is reachable before submit
// (host and aux-service shuffle port of the nodes that will serve map outputs)
try (java.net.Socket s = new java.net.Socket()) {
  s.connect(new java.net.InetSocketAddress(nmHost, shufflePort), 5000);
} catch (java.io.IOException e) {
  throw new IllegalStateException("Shuffle endpoint unreachable: " + nmHost + ":" + shufflePort, e);
}

catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable c = e.getCause(); if (c instanceof java.io.IOException && String.valueOf(c.getMessage()).contains("Got invalid response code")) { /* host/proxy issue: log host from message, fail over or resubmit */ } else { throw e; } }

When it happens

Trigger: Fetcher.copyMapOutput() opens the map output URL and readSslHeaderCode?/shuffle gets rc != HttpURLConnection.HTTP_OK: 400 for a stale/out-of-range map attempt id, 404 when the map output spill was deleted, 401/403 when the job-token handshake is rejected, 500/502/503 when the NodeManager shuffle handler throws, or any non-200 produced by an intermediate HTTP proxy answering instead of the shuffle service.

Common situations: NodeManager restarted or its local dirs were cleaned while reducers were still fetching; a proxy or SSL terminator in the shuffle path that does not forward to the mapreduce_shuffle aux-service port; version or security-config skew between the job client and the cluster; NM disk failures making spill files unreadable.

Related errors

• read past end of stream reading
• failures downloading
• Checksum error reading spill index: " + indexFileName
• error in shuffle in
• Error while doing final merge

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8ef2b637cfb0862b. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java:454

  private void verifyConnection(URL url, String msgToEncode, String encHash)
      throws IOException {
    // Validate response code
    int rc = connection.getResponseCode();
    // See if the shuffleHandler rejected the connection due to too many
    // reducer requests. If so, signal fetchers to back off.
    if (rc == TOO_MANY_REQ_STATUS_CODE) {
      long backoff = connection.getHeaderFieldLong(FETCH_RETRY_AFTER_HEADER,
          FETCH_RETRY_DELAY_DEFAULT);
      // in case we get a negative backoff from ShuffleHandler
      if (backoff < 0) {
        backoff = FETCH_RETRY_DELAY_DEFAULT;
        LOG.warn("Get a negative backoff value from ShuffleHandler. Setting" +
            " it to the default value " + FETCH_RETRY_DELAY_DEFAULT);
      }
      throw new TryAgainLaterException(backoff, url.getHost());
    }
    if (rc != HttpURLConnection.HTTP_OK) {
      throw new IOException(
          "Got invalid response code " + rc + " from " + url +
          ": " + connection.getResponseMessage());
    }
    // get the shuffle version
    if (!ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(
        connection.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))
        || !ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(
            connection.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION))) {
      throw new IOException("Incompatible shuffle response version");
    }
    // get the replyHash which is HMac of the encHash we sent to the server
    String replyHash = connection.getHeaderField(SecureShuffleUtils.HTTP_HEADER_REPLY_URL_HASH);
    if(replyHash==null) {
      throw new IOException("security validation of TT Map output failed");
    }
    LOG.debug("url="+msgToEncode+";encHash="+encHash+";replyHash="+replyHash);
    // verify that replyHash is HMac of encHash
    SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey);

View on GitHub (pinned to 2add963021)