{"record":{"id":"eb759eff0f0e8a07","repo":"apache/hadoop","slug":"verification-of-the-hashreply-failed","errorCode":null,"errorMessage":"Verification of the hashReply failed","messagePattern":"Verification of the hashReply failed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SecureShuffleUtils.java","lineNumber":105,"sourceCode":"  public static String hashFromString(String enc_str, SecretKey key) \n  throws IOException {\n    return generateHash(enc_str.getBytes(StandardCharsets.UTF_8), key);\n  }\n  \n  /**\n   * verify that base64Hash is same as HMacHash(msg)  \n   * @param base64Hash (Base64 encoded hash)\n   * @param msg\n   * @throws IOException if not the same\n   */\n  public static void verifyReply(String base64Hash, String msg, SecretKey key)\n  throws IOException {\n    byte[] hash = Base64.decodeBase64(base64Hash.getBytes(StandardCharsets.UTF_8));\n    \n    boolean res = verifyHash(hash, msg.getBytes(StandardCharsets.UTF_8), key);\n    \n    if(res != true) {\n      throw new IOException(\"Verification of the hashReply failed\");\n    }\n  }\n  \n  /**\n   * Shuffle specific utils - build string for encoding from URL\n   * @param url\n   * @return string for encoding\n   */\n  public static String buildMsgFrom(URL url) {\n    return buildMsgFrom(url.getPath(), url.getQuery(), url.getPort());\n  }\n  /**\n   * Shuffle specific utils - build string for encoding from URL\n   * @param request\n   * @return string for encoding\n   */\n  public static String buildMsgFrom(HttpServletRequest request ) {\n    return buildMsgFrom(request.getRequestURI(), request.getQueryString(),","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SecureShuffleUtils.java#L87-L123","documentation":"During shuffle, the reducer verifies the shuffle handler's reply: SecureShuffleUtils.verifyReply base64-decodes the hash header and recomputes an HMAC-SHA1 over the msg (built from URL path+query+port) using the job-token shuffle secret. A mismatch throws this IOException — the message bytes or the secret differ between the two sides, so the reply cannot be trusted.","triggerScenarios":"An intermediate proxy/router rewrites the shuffle URL (path, query, or port changed after the hash was computed); the job-token shuffle secret is not identical on fetcher and server (token not localized); a custom ShuffleHandler builds the hash over a different message; mixed Hadoop versions whose buildMsgFrom differs.","commonSituations":"HTTP proxies or routers between reducers and NodeManager shuffle ports; job token localization failures on the NM; clusters upgraded node-by-node so client and shuffle handler disagree on the hashed string; custom shuffle plugins that hash only the path.","solutions":["Bypass any proxy that rewrites shuffle URLs — reducers must reach the NM shuffle port directly","Verify the job token secret matches both sides: check jobtoken file localization in NM logs and job credentials propagation","For custom shuffle plugins, hash exactly buildMsgFrom(path, query, port) with the same UTF-8 encoding on both ends","Run identical Hadoop versions on the nodes serving and fetching shuffle data"],"exampleFix":"// before (custom handler): hash over path only\nbyte[] hash = SecureShuffleUtils.generateHash(url.getPath().getBytes(), shuffleSecret);\n// after: both sides hash the same msg\nString msg = SecureShuffleUtils.buildMsgFrom(url);\nbyte[] hash = SecureShuffleUtils.generateHash(msg.getBytes(StandardCharsets.UTF_8), shuffleSecret);","handlingStrategy":"try-catch","validationCode":"String msg = SecureShuffleUtils.buildMsgFrom(url);\n// both sides must derive msg the same way before hashing/verifying","typeGuard":null,"tryCatchPattern":"try {\n  SecureShuffleUtils.verifyReply(replyHash, msg, shuffleKey);\n} catch (IOException e) {\n  throw new IOException(\"Shuffle reply HMAC mismatch for \" + msg\n      + \" — check proxy URL rewriting and job token propagation\", e);\n}","preventionTips":["Never place an HTTP proxy that rewrites URLs or ports between reducers and NM shuffle ports","Propagate the job token to every shuffle participant; verify localization in NM logs","Pin one Hadoop version across the cluster when shuffle is in flight"],"tags":["mapreduce","shuffle","security","hmac","job-token"],"backgroundTag":"hmac-verification-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}