apache/hadoop · error · IOException

Incompatible shuffle response version

Error message

Incompatible shuffle response version

What it means

After a 200 response the Fetcher requires the response to carry the shuffle protocol headers (ShuffleHeader.HTTP_HEADER_NAME 'mapreduce.shuffle' and HTTP_HEADER_VERSION '1.0.0'). Missing or unexpected values mean the endpoint speaks HTTP but is not a MapReduce shuffle handler of the expected protocol version, so the fetch fails with IOException.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java:463

      // in case we get a negative backoff from ShuffleHandler
      if (backoff < 0) {
        backoff = FETCH_RETRY_DELAY_DEFAULT;
        LOG.warn("Get a negative backoff value from ShuffleHandler. Setting" +
            " it to the default value " + FETCH_RETRY_DELAY_DEFAULT);
      }
      throw new TryAgainLaterException(backoff, url.getHost());
    }
    if (rc != HttpURLConnection.HTTP_OK) {
      throw new IOException(
          "Got invalid response code " + rc + " from " + url +
          ": " + connection.getResponseMessage());
    }
    // get the shuffle version
    if (!ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(
        connection.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))
        || !ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(
            connection.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION))) {
      throw new IOException("Incompatible shuffle response version");
    }
    // get the replyHash which is HMac of the encHash we sent to the server
    String replyHash = connection.getHeaderField(SecureShuffleUtils.HTTP_HEADER_REPLY_URL_HASH);
    if(replyHash==null) {
      throw new IOException("security validation of TT Map output failed");
    }
    LOG.debug("url="+msgToEncode+";encHash="+encHash+";replyHash="+replyHash);
    // verify that replyHash is HMac of encHash
    SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey);
    LOG.debug("for url="+msgToEncode+" sent hash and received reply");
  }

  private void setupShuffleConnection(String encHash) {
    // put url hash into http header
    connection.addRequestProperty(
        SecureShuffleUtils.HTTP_HEADER_URL_HASH, encHash);
    // set the read timeout
    connection.setReadTimeout(readTimeout);

View on GitHub (pinned to 2add963021)

Solutions

1. Verify every NodeManager has mapreduce_shuffle in yarn.nodemanager.aux-services and was restarted after the change.
2. Confirm the shuffle URL host:port is the aux-service shuffle port, not the NM webapp or another service.
3. Align the Hadoop/MR version of the job submission environment with the cluster.
4. Remove or fix intermediary proxies that strip custom headers from the shuffle response.

Example fix

# before: aux service not loaded on a NodeManager, its port answers with plain HTTP 200
# yarn-site.xml on that node lacks the service:
#   <name>yarn.nodemanager.aux-services</name><value></value>
# after: declare it and restart the NodeManager
#   <name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>
yarn --daemon stop nodemanager && yarn --daemon start nodemanager

// Probe a shuffle endpoint for the protocol headers the Fetcher will require
import org.apache.hadoop.mapreduce.task.reduce.ShuffleHeader;
java.net.HttpURLConnection c = (java.net.HttpURLConnection) new java.net.URL(url).openConnection();
boolean ok = ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(c.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))
    && ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(c.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION));

catch (java.io.IOException e) { if ("Incompatible shuffle response version".equals(e.getMessage())) { /* wrong endpoint or version skew: verify aux-services before retrying */ } else { throw e; } }

When it happens

Trigger: The fetch URL resolves to a service that is not the mapreduce_shuffle auxiliary service (plain Jetty endpoint, wrong port, Spark shuffle service); a version skew between the MR client and the NodeManager; a proxy that strips or renames custom response headers.

Common situations: yarn.nodemanager.aux-services missing mapreduce_shuffle on some NodeManagers (often after a config change without restart); pointing reducers at the NM webapp port instead of the aux-service shuffle port; mixed-version clusters during rolling upgrades.

Related errors

• Serverside implements {}. The following requested protocol i
• Serverside implements {}. The following requested protocol i
• RPC response length mismatch
• Unknown protocol: {}
• Unrecognized priority: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/990891dddda571c4. Report an issue: GitHub.