apache/hadoop · error · IOException

Failed to complete SASL handshake

Error message

Failed to complete SASL handshake

What it means

On a secured cluster (Kerberos or DIGEST-MD5 token authentication with data transfer protection), checkSaslComplete() verifies after SASL negotiation that the SaslParticipant actually completed the handshake. isComplete()==false means the SASL mechanism aborted mid-exchange - credentials were rejected (invalid or expired token/password) or the socket was cut - so the data connection cannot be used and an IOException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java:100

  /**
   * Sent by clients and validated by servers. We use a number that's unlikely
   * to ever be sent as the value of the DATA_TRANSFER_VERSION.
   */
  public static final int SASL_TRANSFER_MAGIC_NUMBER = 0xDEADBEEF;

  /**
   * Checks that SASL negotiation has completed for the given participant, and
   * the negotiated quality of protection is included in the given SASL
   * properties and therefore acceptable.
   *
   * @param sasl participant to check
   * @param saslProps properties of SASL negotiation
   * @throws IOException for any error
   */
  public static void checkSaslComplete(SaslParticipant sasl,
      Map<String, String> saslProps) throws IOException {
    if (!sasl.isComplete()) {
      throw new IOException("Failed to complete SASL handshake");
    }
    Set<String> requestedQop = ImmutableSet.copyOf(Arrays.asList(
        saslProps.get(Sasl.QOP).split(",")));
    String negotiatedQop = sasl.getNegotiatedQop();
    LOG.debug("{}: Verifying QOP: requested = {}, negotiated = {}",
        sasl, requestedQop, negotiatedQop);
    // Treat null negotiated QOP as "auth" for the purpose of verification
    // Code elsewhere does the same implicitly
    if(negotiatedQop == null) {
      negotiatedQop = "auth";
    }
    if (!requestedQop.contains(negotiatedQop)) {
      throw new IOException(String.format("SASL handshake completed, but " +
          "channel does not have acceptable quality of protection, " +
          "requested = %s, negotiated(effective) = %s", requestedQop, negotiatedQop));
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Check DataNode logs - the underlying SASL error (rejected credential) is only visible there in detail
2. Refresh credentials: re-login via keytab or obtain a new delegation token, then retry the operation
3. Make dfs.data.transfer.protection identical on NameNode, DataNodes and client, and restart/reload configs
4. Align Hadoop versions between client and DataNodes if the negotiation sequence differs

try {
  dfsDataOperation();
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("Failed to complete SASL handshake")) {
    // credentials rejected mid-negotiation: refresh token / re-login, retry once
    refreshCredentialsAndRetry();
  } else {
    throw e;
  }
}

When it happens

Trigger: SaslDataTransferClient/Server negotiate a protected data connection; the peer SASL server returns an error or closes the connection during the challenge/response steps; checkSaslComplete(sasl, saslProps) then finds !sasl.isComplete().

Common situations: Expired or non-renewed delegation tokens on long-running jobs; token password mismatch after failover or token re-creation; Kerberos/keytab problems on the DataNode; interrupted sockets during rolling upgrades.

Understand the failure class

• SSL/TLS and certificate errors — how TLS handshakes and certificate validation fail.

Related errors

• Server asks us to fall back to SIMPLE auth, but this client
• Can't retrieve username from tokenIdentifier.
• ${method} authentication is not enabled. Available:${enable
• Cannot create a secured connection if DataNode listens on un
• Security enabled but user not authenticated by filter

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/60ab86795fb018bf. Report an issue: GitHub.