apache/hadoop · error · IOException

Got invalid response code {rc} from {url}: {responseMessage}

Error message

Got invalid response code {rc} from {url}: {responseMessage}

What it means

Thrown by the reduce-side Fetcher when an HTTP shuffle request to a NodeManager's shuffle handler returns a status other than 200 OK (and other than 429, which is converted to TryAgainLaterException honoring the Retry-After header). This is a per-map-output fetch failure: ShuffleSchedulerImpl penalizes the host with an exponential backoff and re-queues the output, so the reducer only dies if the failure repeats past abortFailureLimit.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java:454

  private void verifyConnection(URL url, String msgToEncode, String encHash)
      throws IOException {
    // Validate response code
    int rc = connection.getResponseCode();
    // See if the shuffleHandler rejected the connection due to too many
    // reducer requests. If so, signal fetchers to back off.
    if (rc == TOO_MANY_REQ_STATUS_CODE) {
      long backoff = connection.getHeaderFieldLong(FETCH_RETRY_AFTER_HEADER,
          FETCH_RETRY_DELAY_DEFAULT);
      // in case we get a negative backoff from ShuffleHandler
      if (backoff < 0) {
        backoff = FETCH_RETRY_DELAY_DEFAULT;
        LOG.warn("Get a negative backoff value from ShuffleHandler. Setting" +
            " it to the default value " + FETCH_RETRY_DELAY_DEFAULT);
      }
      throw new TryAgainLaterException(backoff, url.getHost());
    }
    if (rc != HttpURLConnection.HTTP_OK) {
      throw new IOException(
          "Got invalid response code " + rc + " from " + url +
          ": " + connection.getResponseMessage());
    }
    // get the shuffle version
    if (!ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(
        connection.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))
        || !ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(
            connection.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION))) {
      throw new IOException("Incompatible shuffle response version");
    }
    // get the replyHash which is HMac of the encHash we sent to the server
    String replyHash = connection.getHeaderField(SecureShuffleUtils.HTTP_HEADER_REPLY_URL_HASH);
    if(replyHash==null) {
      throw new IOException("security validation of TT Map output failed");
    }
    LOG.debug("url="+msgToEncode+";encHash="+encHash+";replyHash="+replyHash);
    // verify that replyHash is HMac of encHash
    SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey);

View on GitHub (pinned to 2add963021)

Solutions

1. Correlate with the NodeManager log on the host named in the URL at the same timestamp; the server-side stack trace explains the non-200 code.
2. If 404/400: the map attempt is stale on that node; the framework re-fetches another attempt, so verify the AM re-ran the map and that failure tolerance covers the window.
3. If 401/403: check job-token propagation and that client and cluster share the same hadoop.security.authentication setup.
4. If a proxy sits in the shuffle path, remove it or make it pass the NodeManager shuffle aux-service port through without rewriting the response.

Example fix

// before: large map outputs time out server-side and yield 5xx; small read timeout
conf.setInt("mapreduce.reduce.shuffle.read.timeout", 80000);
// after: give slow NodeManagers time to stream big spills
conf.setInt("mapreduce.reduce.shuffle.read.timeout", 240000);
conf.setInt("mapreduce.reduce.shuffle.connect.timeout", 180000);

// Pre-flight: verify the NodeManager shuffle endpoint is reachable before submit
// (host and aux-service shuffle port of the nodes that will serve map outputs)
try (java.net.Socket s = new java.net.Socket()) {
  s.connect(new java.net.InetSocketAddress(nmHost, shufflePort), 5000);
} catch (java.io.IOException e) {
  throw new IllegalStateException("Shuffle endpoint unreachable: " + nmHost + ":" + shufflePort, e);
}

catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable c = e.getCause(); if (c instanceof java.io.IOException && String.valueOf(c.getMessage()).contains("Got invalid response code")) { /* host/proxy issue: log host from message, fail over or resubmit */ } else { throw e; } }

When it happens

Trigger: Fetcher.copyMapOutput() opens the map output URL and readSslHeaderCode?/shuffle gets rc != HttpURLConnection.HTTP_OK: 400 for a stale/out-of-range map attempt id, 404 when the map output spill was deleted, 401/403 when the job-token handshake is rejected, 500/502/503 when the NodeManager shuffle handler throws, or any non-200 produced by an intermediate HTTP proxy answering instead of the shuffle service.

Common situations: NodeManager restarted or its local dirs were cleaned while reducers were still fetching; a proxy or SSL terminator in the shuffle path that does not forward to the mapreduce_shuffle aux-service port; version or security-config skew between the job client and the cluster; NM disk failures making spill files unreadable.

Related errors

• read past end of stream reading {mapId}
• {failures} failures downloading {mapId}
• Checksum error reading spill index: " + indexFileName
• server didn't return all expected map outputs: {remaining.si
• Incompatible shuffle response version

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8ef2b637cfb0862b. Report an issue: GitHub.