apache/hadoop · critical · FatalRpcServerException

FATAL_INVALID_RPC_HEADER

FATAL_INVALID_RPC_HEADER

Error message

Server is not wrapping data

What it means

saslReadAndProcess received an SASL WRAP frame (wrapped RPC payload) while the server either had not completed SASL negotiation (saslContextEstablished false) or negotiated a QOP without wrapping (useWrap false, e.g., auth-only). It responds with FatalRpcServerException using error code FATAL_INVALID_RPC_HEADER — the connection is terminated because client and server disagree on SASL state or protection level.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:2252

        if (ugi == null) {
          throw new AccessControlException(
              "Can't retrieve username from tokenIdentifier.");
        }
        ugi.addTokenIdentifier(tokenId);
        return ugi;
      } else {
        return UserGroupInformation.createRemoteUser(authorizedId, authMethod);
      }
    }

    private void saslReadAndProcess(RpcWritable.Buffer buffer) throws
        RpcServerException, IOException, InterruptedException {
      final RpcSaslProto saslMessage =
          getMessage(RpcSaslProto.getDefaultInstance(), buffer);
      switch (saslMessage.getState()) {
        case WRAP: {
          if (!saslContextEstablished || !useWrap) {
            throw new FatalRpcServerException(
                RpcErrorCodeProto.FATAL_INVALID_RPC_HEADER,
                new SaslException("Server is not wrapping data"));
          }
          // loops over decoded data and calls processOneRpc
          unwrapPacketAndProcessRpcs(saslMessage.getToken().toByteArray());
          break;
        }
        default:
          saslProcess(saslMessage);
      }
    }

    /**
     * Some exceptions ({@link RetriableException} and {@link StandbyException})
     * that are wrapped as a cause of parameter e are unwrapped so that they can
     * be sent as the true cause to the client side. In case of
     * {@link InvalidToken} we go one level deeper to get the true cause.
     * 

View on GitHub (pinned to 2add963021)

Solutions

1. Align hadoop.rpc.protection on client and server — the server's allowed level must cover the client's requested level; setting both to the same value is simplest.
2. Use the standard RPC client (RPC.getProxy, DFSClient, YARN clients) rather than hand-driving the SASL exchange.
3. If wrapping is genuinely required, enable integrity/privacy on the server so useWrap is true.

Example fix

<!-- before: client wraps, server does not -->
<!-- client core-site.xml -->
<property><name>hadoop.rpc.protection</name><value>privacy</value></property>
<!-- after: both sides agree -->
<property><name>hadoop.rpc.protection</name><value>authentication</value></property> <!-- same on server -->

static int rank(String p) { // authentication < integrity < privacy
  switch (p) {
    case "integrity": return 1;
    case "privacy": return 2;
    default: return 0;
  }
}
String client = conf.get("hadoop.rpc.protection", "authentication");
String server = serverConf.get("hadoop.rpc.protection", "authentication");
if (rank(client) > rank(server)) {
  throw new IllegalStateException(
      "client QOP '" + client + "' exceeds server allowance '" + server + "'");
}

When it happens

Trigger: A client sending wrapped payloads before SASL completes; a client assuming integrity/privacy wrapping (hadoop.rpc.protection=privacy or integrity) against a server configured for authentication only; hand-written clients driving the SASL state machine out of order.

Common situations: hadoop.rpc.protection mismatch between client and server core-site.xml (client privacy, server authentication); cross-cluster access such as distcp between clusters with different protection settings; custom or native clients implementing SASL themselves.

Related errors

• Can't retrieve username from tokenIdentifier.
• Client already attempted negotiation
• Client mechanism is malformed
• ${method} authentication is not enabled. Available:${enable
• Client sent unsupported state ${state}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ba40c87b3b5f762a. Report an issue: GitHub.