{"record":{"id":"990891dddda571c4","repo":"apache/hadoop","slug":"incompatible-shuffle-response-version","errorCode":null,"errorMessage":"Incompatible shuffle response version","messagePattern":"Incompatible shuffle response version","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java","lineNumber":463,"sourceCode":"      // in case we get a negative backoff from ShuffleHandler\n      if (backoff < 0) {\n        backoff = FETCH_RETRY_DELAY_DEFAULT;\n        LOG.warn(\"Get a negative backoff value from ShuffleHandler. Setting\" +\n            \" it to the default value \" + FETCH_RETRY_DELAY_DEFAULT);\n      }\n      throw new TryAgainLaterException(backoff, url.getHost());\n    }\n    if (rc != HttpURLConnection.HTTP_OK) {\n      throw new IOException(\n          \"Got invalid response code \" + rc + \" from \" + url +\n          \": \" + connection.getResponseMessage());\n    }\n    // get the shuffle version\n    if (!ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(\n        connection.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))\n        || !ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(\n            connection.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION))) {\n      throw new IOException(\"Incompatible shuffle response version\");\n    }\n    // get the replyHash which is HMac of the encHash we sent to the server\n    String replyHash = connection.getHeaderField(SecureShuffleUtils.HTTP_HEADER_REPLY_URL_HASH);\n    if(replyHash==null) {\n      throw new IOException(\"security validation of TT Map output failed\");\n    }\n    LOG.debug(\"url=\"+msgToEncode+\";encHash=\"+encHash+\";replyHash=\"+replyHash);\n    // verify that replyHash is HMac of encHash\n    SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey);\n    LOG.debug(\"for url=\"+msgToEncode+\" sent hash and received reply\");\n  }\n\n  private void setupShuffleConnection(String encHash) {\n    // put url hash into http header\n    connection.addRequestProperty(\n        SecureShuffleUtils.HTTP_HEADER_URL_HASH, encHash);\n    // set the read timeout\n    connection.setReadTimeout(readTimeout);","sourceCodeStart":445,"sourceCodeEnd":481,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java#L445-L481","documentation":"After a 200 response the Fetcher requires the response to carry the shuffle protocol headers (ShuffleHeader.HTTP_HEADER_NAME 'mapreduce.shuffle' and HTTP_HEADER_VERSION '1.0.0'). Missing or unexpected values mean the endpoint speaks HTTP but is not a MapReduce shuffle handler of the expected protocol version, so the fetch fails with IOException.","triggerScenarios":"The fetch URL resolves to a service that is not the mapreduce_shuffle auxiliary service (plain Jetty endpoint, wrong port, Spark shuffle service); a version skew between the MR client and the NodeManager; a proxy that strips or renames custom response headers.","commonSituations":"yarn.nodemanager.aux-services missing mapreduce_shuffle on some NodeManagers (often after a config change without restart); pointing reducers at the NM webapp port instead of the aux-service shuffle port; mixed-version clusters during rolling upgrades.","solutions":["Verify every NodeManager has mapreduce_shuffle in yarn.nodemanager.aux-services and was restarted after the change.","Confirm the shuffle URL host:port is the aux-service shuffle port, not the NM webapp or another service.","Align the Hadoop/MR version of the job submission environment with the cluster.","Remove or fix intermediary proxies that strip custom headers from the shuffle response."],"exampleFix":"# before: aux service not loaded on a NodeManager, its port answers with plain HTTP 200\n# yarn-site.xml on that node lacks the service:\n#   <name>yarn.nodemanager.aux-services</name><value></value>\n# after: declare it and restart the NodeManager\n#   <name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>\nyarn --daemon stop nodemanager && yarn --daemon start nodemanager","handlingStrategy":"validation","validationCode":"// Probe a shuffle endpoint for the protocol headers the Fetcher will require\nimport org.apache.hadoop.mapreduce.task.reduce.ShuffleHeader;\njava.net.HttpURLConnection c = (java.net.HttpURLConnection) new java.net.URL(url).openConnection();\nboolean ok = ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(c.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))\n    && ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(c.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION));","typeGuard":null,"tryCatchPattern":"catch (java.io.IOException e) { if (\"Incompatible shuffle response version\".equals(e.getMessage())) { /* wrong endpoint or version skew: verify aux-services before retrying */ } else { throw e; } }","preventionTips":["Declare mapreduce_shuffle in yarn.nodemanager.aux-services on every NodeManager and restart after changes.","Smoke-test one shuffle fetch after cluster or version upgrades before launching large jobs.","Do not point reducers at the NM webapp port or foreign shuffle services."],"tags":["hadoop","mapreduce","shuffle","protocol","version-mismatch","http-headers"],"backgroundTag":"protocol-version-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}