{"record":{"id":"8ef2b637cfb0862b","repo":"apache/hadoop","slug":"got-invalid-response-code-rc-from-url-respon","errorCode":null,"errorMessage":"Got invalid response code {rc} from {url}: {responseMessage}","messagePattern":"Got invalid response code (.+?) from (.+?): (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java","lineNumber":454,"sourceCode":"  private void verifyConnection(URL url, String msgToEncode, String encHash)\n      throws IOException {\n    // Validate response code\n    int rc = connection.getResponseCode();\n    // See if the shuffleHandler rejected the connection due to too many\n    // reducer requests. If so, signal fetchers to back off.\n    if (rc == TOO_MANY_REQ_STATUS_CODE) {\n      long backoff = connection.getHeaderFieldLong(FETCH_RETRY_AFTER_HEADER,\n          FETCH_RETRY_DELAY_DEFAULT);\n      // in case we get a negative backoff from ShuffleHandler\n      if (backoff < 0) {\n        backoff = FETCH_RETRY_DELAY_DEFAULT;\n        LOG.warn(\"Get a negative backoff value from ShuffleHandler. Setting\" +\n            \" it to the default value \" + FETCH_RETRY_DELAY_DEFAULT);\n      }\n      throw new TryAgainLaterException(backoff, url.getHost());\n    }\n    if (rc != HttpURLConnection.HTTP_OK) {\n      throw new IOException(\n          \"Got invalid response code \" + rc + \" from \" + url +\n          \": \" + connection.getResponseMessage());\n    }\n    // get the shuffle version\n    if (!ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(\n        connection.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))\n        || !ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(\n            connection.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION))) {\n      throw new IOException(\"Incompatible shuffle response version\");\n    }\n    // get the replyHash which is HMac of the encHash we sent to the server\n    String replyHash = connection.getHeaderField(SecureShuffleUtils.HTTP_HEADER_REPLY_URL_HASH);\n    if(replyHash==null) {\n      throw new IOException(\"security validation of TT Map output failed\");\n    }\n    LOG.debug(\"url=\"+msgToEncode+\";encHash=\"+encHash+\";replyHash=\"+replyHash);\n    // verify that replyHash is HMac of encHash\n    SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey);","sourceCodeStart":436,"sourceCodeEnd":472,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java#L436-L472","documentation":"Thrown by the reduce-side Fetcher when an HTTP shuffle request to a NodeManager's shuffle handler returns a status other than 200 OK (and other than 429, which is converted to TryAgainLaterException honoring the Retry-After header). This is a per-map-output fetch failure: ShuffleSchedulerImpl penalizes the host with an exponential backoff and re-queues the output, so the reducer only dies if the failure repeats past abortFailureLimit.","triggerScenarios":"Fetcher.copyMapOutput() opens the map output URL and readSslHeaderCode?/shuffle gets rc != HttpURLConnection.HTTP_OK: 400 for a stale/out-of-range map attempt id, 404 when the map output spill was deleted, 401/403 when the job-token handshake is rejected, 500/502/503 when the NodeManager shuffle handler throws, or any non-200 produced by an intermediate HTTP proxy answering instead of the shuffle service.","commonSituations":"NodeManager restarted or its local dirs were cleaned while reducers were still fetching; a proxy or SSL terminator in the shuffle path that does not forward to the mapreduce_shuffle aux-service port; version or security-config skew between the job client and the cluster; NM disk failures making spill files unreadable.","solutions":["Correlate with the NodeManager log on the host named in the URL at the same timestamp; the server-side stack trace explains the non-200 code.","If 404/400: the map attempt is stale on that node; the framework re-fetches another attempt, so verify the AM re-ran the map and that failure tolerance covers the window.","If 401/403: check job-token propagation and that client and cluster share the same hadoop.security.authentication setup.","If a proxy sits in the shuffle path, remove it or make it pass the NodeManager shuffle aux-service port through without rewriting the response."],"exampleFix":"// before: large map outputs time out server-side and yield 5xx; small read timeout\nconf.setInt(\"mapreduce.reduce.shuffle.read.timeout\", 80000);\n// after: give slow NodeManagers time to stream big spills\nconf.setInt(\"mapreduce.reduce.shuffle.read.timeout\", 240000);\nconf.setInt(\"mapreduce.reduce.shuffle.connect.timeout\", 180000);","handlingStrategy":"retry","validationCode":"// Pre-flight: verify the NodeManager shuffle endpoint is reachable before submit\n// (host and aux-service shuffle port of the nodes that will serve map outputs)\ntry (java.net.Socket s = new java.net.Socket()) {\n  s.connect(new java.net.InetSocketAddress(nmHost, shufflePort), 5000);\n} catch (java.io.IOException e) {\n  throw new IllegalStateException(\"Shuffle endpoint unreachable: \" + nmHost + \":\" + shufflePort, e);\n}","typeGuard":null,"tryCatchPattern":"catch (org.apache.hadoop.mapreduce.task.reduce.Shuffle.ShuffleError e) { Throwable c = e.getCause(); if (c instanceof java.io.IOException && String.valueOf(c.getMessage()).contains(\"Got invalid response code\")) { /* host/proxy issue: log host from message, fail over or resubmit */ } else { throw e; } }","preventionTips":["Keep NodeManager local dirs healthy and monitored so spill files survive until all reducers fetch.","Never put header-rewriting proxies in the shuffle path.","Keep the job client's Hadoop version aligned with the cluster.","Set mapreduce.reduce.shuffle.connect/read timeouts sized for your largest map outputs."],"tags":["hadoop","mapreduce","shuffle","fetcher","http","network"],"backgroundTag":"http-error-status-code","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}