apache/hadoop · error · IOException

security validation of TT Map output failed

Error message

security validation of TT Map output failed

What it means

With secure shuffle the reducer sends an HMAC of the request URL (encHash) and expects the server to echo a reply hash header (SecureShuffleUtils.HTTP_HEADER_REPLY_URL_HASH). A null header means the server never performed the secure-shuffle handshake, so the Fetcher fails before SecureShuffleUtils.verifyReply can even compare HMACs.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java:468

      }
      throw new TryAgainLaterException(backoff, url.getHost());
    }
    if (rc != HttpURLConnection.HTTP_OK) {
      throw new IOException(
          "Got invalid response code " + rc + " from " + url +
          ": " + connection.getResponseMessage());
    }
    // get the shuffle version
    if (!ShuffleHeader.DEFAULT_HTTP_HEADER_NAME.equals(
        connection.getHeaderField(ShuffleHeader.HTTP_HEADER_NAME))
        || !ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION.equals(
            connection.getHeaderField(ShuffleHeader.HTTP_HEADER_VERSION))) {
      throw new IOException("Incompatible shuffle response version");
    }
    // get the replyHash which is HMac of the encHash we sent to the server
    String replyHash = connection.getHeaderField(SecureShuffleUtils.HTTP_HEADER_REPLY_URL_HASH);
    if(replyHash==null) {
      throw new IOException("security validation of TT Map output failed");
    }
    LOG.debug("url="+msgToEncode+";encHash="+encHash+";replyHash="+replyHash);
    // verify that replyHash is HMac of encHash
    SecureShuffleUtils.verifyReply(replyHash, encHash, shuffleSecretKey);
    LOG.debug("for url="+msgToEncode+" sent hash and received reply");
  }

  private void setupShuffleConnection(String encHash) {
    // put url hash into http header
    connection.addRequestProperty(
        SecureShuffleUtils.HTTP_HEADER_URL_HASH, encHash);
    // set the read timeout
    connection.setReadTimeout(readTimeout);
    // put shuffle version into http header
    connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_NAME,
        ShuffleHeader.DEFAULT_HTTP_HEADER_NAME);
    connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_VERSION,
        ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION);

View on GitHub (pinned to 2add963021)

Solutions

1. Check the NodeManager log on the source host for handshake/token errors for this app attempt.
2. Verify no proxy in the path removes custom response headers; the reply-hash header must pass through untouched.
3. Confirm client and cluster run compatible Hadoop versions and the same security settings.
4. Restart the NodeManager if it lost the application's shuffle secret, then let the job retry.

catch (java.io.IOException e) { if ("security validation of TT Map output failed".equals(e.getMessage())) { /* token/proxy issue: restart NM or fix header passthrough, then let the scheduler retry */ } else { throw e; } }

When it happens

Trigger: The NodeManager shuffle handler served the response without computing the reply hash because it lacks the job token / shuffle secret for this app attempt; an intermediate proxy stripped the reply header; version skew where the server side does not implement the secure handshake.

Common situations: Aux service not initialized for the application (NM restarted mid-job and lost app state); restrictive proxies or firewalls rewriting headers; mixed Hadoop versions during upgrades; job-token files unreadable due to NM local-dir permission problems.

Related errors

• Verification of the hashReply failed
• Checksum error reading spill index: " + indexFileName
• Got invalid response code {rc} from {url}: {responseMessage}
• Incompatible shuffle response version
• Invalid timeout [timeout = {connectionTimeout} ms]

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bbb9a3f204e3451b. Report an issue: GitHub.