What causes "Verification of the hashReply failed" in apache/hadoop?

Bypass any proxy that rewrites shuffle URLs — reducers must reach the NM shuffle port directly

apache/hadoop · error · IOException

Verification of the hashReply failed

Error message

Verification of the hashReply failed

What it means

During shuffle, the reducer verifies the shuffle handler's reply: SecureShuffleUtils.verifyReply base64-decodes the hash header and recomputes an HMAC-SHA1 over the msg (built from URL path+query+port) using the job-token shuffle secret. A mismatch throws this IOException — the message bytes or the secret differ between the two sides, so the reply cannot be trusted.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SecureShuffleUtils.java:105

  public static String hashFromString(String enc_str, SecretKey key) 
  throws IOException {
    return generateHash(enc_str.getBytes(StandardCharsets.UTF_8), key);
  }
  
  /**
   * verify that base64Hash is same as HMacHash(msg)  
   * @param base64Hash (Base64 encoded hash)
   * @param msg
   * @throws IOException if not the same
   */
  public static void verifyReply(String base64Hash, String msg, SecretKey key)
  throws IOException {
    byte[] hash = Base64.decodeBase64(base64Hash.getBytes(StandardCharsets.UTF_8));
    
    boolean res = verifyHash(hash, msg.getBytes(StandardCharsets.UTF_8), key);
    
    if(res != true) {
      throw new IOException("Verification of the hashReply failed");
    }
  }
  
  /**
   * Shuffle specific utils - build string for encoding from URL
   * @param url
   * @return string for encoding
   */
  public static String buildMsgFrom(URL url) {
    return buildMsgFrom(url.getPath(), url.getQuery(), url.getPort());
  }
  /**
   * Shuffle specific utils - build string for encoding from URL
   * @param request
   * @return string for encoding
   */
  public static String buildMsgFrom(HttpServletRequest request ) {
    return buildMsgFrom(request.getRequestURI(), request.getQueryString(),

View on GitHub (pinned to 2add963021)

Solutions

Bypass any proxy that rewrites shuffle URLs — reducers must reach the NM shuffle port directly
Verify the job token secret matches both sides: check jobtoken file localization in NM logs and job credentials propagation
For custom shuffle plugins, hash exactly buildMsgFrom(path, query, port) with the same UTF-8 encoding on both ends
Run identical Hadoop versions on the nodes serving and fetching shuffle data

Example fix

// before (custom handler): hash over path only
byte[] hash = SecureShuffleUtils.generateHash(url.getPath().getBytes(), shuffleSecret);
// after: both sides hash the same msg
String msg = SecureShuffleUtils.buildMsgFrom(url);
byte[] hash = SecureShuffleUtils.generateHash(msg.getBytes(StandardCharsets.UTF_8), shuffleSecret);

Defensive patterns

Strategy: try-catch

Validate before calling

String msg = SecureShuffleUtils.buildMsgFrom(url);
// both sides must derive msg the same way before hashing/verifying

Try / catch

try {
  SecureShuffleUtils.verifyReply(replyHash, msg, shuffleKey);
} catch (IOException e) {
  throw new IOException("Shuffle reply HMAC mismatch for " + msg
      + " — check proxy URL rewriting and job token propagation", e);
}

Prevention

Never place an HTTP proxy that rewrites URLs or ports between reducers and NM shuffle ports
Propagate the job token to every shuffle participant; verify localization in NM logs
Pin one Hadoop version across the cluster when shuffle is in flight

When it happens

Trigger: An intermediate proxy/router rewrites the shuffle URL (path, query, or port changed after the hash was computed); the job-token shuffle secret is not identical on fetcher and server (token not localized); a custom ShuffleHandler builds the hash over a different message; mixed Hadoop versions whose buildMsgFrom differs.

Common situations: HTTP proxies or routers between reducers and NodeManager shuffle ports; job token localization failures on the NM; clusters upgraded node-by-node so client and shuffle handler disagree on the hashed string; custom shuffle plugins that hash only the path.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/eb759eff0f0e8a07. Report an issue: GitHub.