apache/hadoop · error · AccessControlException

Disallowed RPC access from {} at {}. Not listed in dfs.clust

Error message

Disallowed RPC access from {} at {}. Not listed in dfs.cluster.administrators

What it means

The ZKFC daemon exposes a small admin RPC server (ZKFCProtocol, contacted for example by 'hdfs haadmin -failover' graceful failover and other admin operations). checkRpcAdminAccess authorizes each call: the caller's UGI must pass adminAcl (built from dfs.cluster.administrators) or the caller's short username must equal the ZKFC daemon's login user. Otherwise the call is refused with AccessControlException and a WARN line naming the rejected ugi and remote address.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java:234

      LOG.error("DFSZKFailOverController exiting due to earlier exception "
          + t);
      terminate(1, t);
    }
  }

  @Override
  protected void checkRpcAdminAccess() throws IOException, AccessControlException {
    UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
    UserGroupInformation zkfcUgi = UserGroupInformation.getLoginUser();
    if (adminAcl.isUserAllowed(ugi) ||
        ugi.getShortUserName().equals(zkfcUgi.getShortUserName())) {
      LOG.info("Allowed RPC access from " + ugi + " at " + Server.getRemoteAddress());
      return;
    }
    String msg = "Disallowed RPC access from " + ugi + " at " +
        Server.getRemoteAddress() + ". Not listed in " + DFSConfigKeys.DFS_ADMIN; 
    LOG.warn(msg);
    throw new AccessControlException(msg);
  }

  /**
   * capture local NN's thread dump and write it to ZKFC's log.
   */
  private void getLocalNNThreadDump() {
    isThreadDumpCaptured = false;
    // We use the same timeout value for both connection establishment
    // timeout and read timeout.
    int httpTimeOut = conf.getInt(
        DFSConfigKeys.DFS_HA_ZKFC_NN_HTTP_TIMEOUT_KEY,
        DFSConfigKeys.DFS_HA_ZKFC_NN_HTTP_TIMEOUT_KEY_DEFAULT);
    if (httpTimeOut == 0) {
      // If timeout value is set to zero, the feature is turned off.
      return;
    }
    try {
      String stacksUrl = DFSUtil.getInfoServer(localNNTarget.getAddress(),

View on GitHub (pinned to 2add963021)

Solutions

Run the admin command as the same user that runs the zkfc daemon (typically hdfs), e.g. 'sudo -u hdfs hdfs haadmin -failover nn1 nn2'.
Add the invoking user or group to dfs.cluster.administrators in hdfs-site.xml (format: comma-separated users, space, comma-separated groups) on the NameNode/zkfc hosts and restart zkfc.
Check the zkfc log line 'Disallowed RPC access from <ugi>' to see exactly which ugi and address the server saw, then align the ACL with it.
Verify with 'hdfs getconf -confKey dfs.cluster.administrators' on the zkfc host that the ACL actually loaded.

Example fix

# before: run as operator, zkfc runs as hdfs
hdfs haadmin -ns mycluster -failover nn1 nn2   # AccessControlException

# after: same user as the zkfc daemon
sudo -u hdfs hdfs haadmin -ns mycluster -failover nn1 nn2

# or in hdfs-site.xml on the zkfc hosts, then restart zkfc
<property>
  <name>dfs.cluster.administrators</name>
  <value>hdfs,opsadmin hadmin</value>
</property>

Defensive patterns

Strategy: try-catch

Validate before calling

// Before issuing graceful failover, confirm the operator passes the ACL
String acl = conf.get("dfs.cluster.administrators", "");
String user = UserGroupInformation.getCurrentUser().getShortUserName();
String zkfcUser = UserGroupInformation.getLoginUser().getShortUserName();
if (!user.equals(zkfcUser) && !aclContains(acl, user)) {
  throw new SecurityException("Run as " + zkfcUser
      + " or add " + user + " to dfs.cluster.administrators");
}

Try / catch

try {
  haAdmin.failover(fromNode, toNode);
} catch (AccessControlException ace) {
  // Message contains 'Not listed in dfs.cluster.administrators'
  throw new UnsupportedOperationException(
      "Failover denied for " + UserGroupInformation.getCurrentUser()
          + ": add user to dfs.cluster.administrators or run as the zkfc daemon user", ace);
}

Prevention

Run failover and other ZKFC admin operations as the same OS user that runs the zkfc daemon.
Maintain dfs.cluster.administrators in config management and reload/restart zkfc after changes.
Include a periodic 'hdfs haadmin -getServiceState' from the monitoring user to surface ACL breaks before a real failover.

When it happens

Trigger: Running 'hdfs haadmin -failover ...' as a user that is neither in the dfs.cluster.administrators ACL nor the same user as the one running the zkfc process; an ACL that lists groups/users with wrong syntax so adminAcl does not match the caller.

Common situations: Operators running failover as their personal account while zkfc runs as hdfs; dfs.cluster.administrators configured on the NameNodes but not picked up by the ZKFC's config; kerberos-authenticated admin whose short name differs from the daemon user.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2a78a73821c23f80. Report an issue: GitHub.