apache/hadoop · error · AccessControlException

Request from ZK failover controller at {Server.getRemoteAddr

Error message

Request from ZK failover controller at {Server.getRemoteAddress()} denied since automatic HA is not enabled

What it means

The REQUEST_BY_ZKFC arm of NameNode.checkHaStateChange: a state-change request sourced from the ZKFailoverController (graceful failover or fencing transitions) is rejected with AccessControlException when this NameNode's conf has dfs.ha.automatic-failover.enabled=false. The NameNode refuses ZKFC-driven transitions it was not configured to accept, and the message names the ZKFC's address via Server.getRemoteAddress(). Note the asymmetry with the REQUEST_BY_USER arm: the same boolean guards both arms in opposite directions.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:2323

    switch (req.getSource()) {
    case REQUEST_BY_USER:
      if (autoHaEnabled) {
        throw new AccessControlException(
            "Manual HA control for this NameNode is disallowed, because " +
            "automatic HA is enabled.");
      }
      break;
    case REQUEST_BY_USER_FORCED:
      if (autoHaEnabled) {
        LOG.warn("Allowing manual HA control from " +
            Server.getRemoteAddress() +
            " even though automatic HA is enabled, because the user " +
            "specified the force flag");
      }
      break;
    case REQUEST_BY_ZKFC:
      if (!autoHaEnabled) {
        throw new AccessControlException(
            "Request from ZK failover controller at " +
            Server.getRemoteAddress() + " denied since automatic HA " +
            "is not enabled"); 
      }
      break;
    }
  }

  /*
   * {@inheritDoc}
   * */
  @Override // ReconfigurableBase
  public Collection<String> getReconfigurableProperties() {
    return reconfigurableProperties;
  }

  /*
   * {@inheritDoc}

View on GitHub (pinned to 2add963021)

Solutions

If automatic failover is no longer wanted, stop the ZKFC daemons (`hdfs --daemon stop zkfc`) so nothing emits REQUEST_BY_ZKFC requests
If automatic failover is wanted, set dfs.ha.automatic-failover.enabled=true in the NameNode's hdfs-site.xml and restart the NameNode
Diff hdfs-site.xml across every HA node and ZKFC host - the flag must agree cluster-wide, since the NN judges only its own copy
After fixing, verify the ZKFC path with `hdfs haadmin -failover` on a test pair

Example fix

<!-- hdfs-site.xml on the NameNode: before -->
<property><name>dfs.ha.automatic-failover.enabled</name><value>false</value></property>

<!-- after (then restart NameNode and ZKFC) -->
<property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property>

Defensive patterns

Strategy: validation

Validate before calling

# Guard: never run ZKFC while the NN disables auto failover
AUTO=$(hdfs getconf -confKey dfs.ha.automatic-failover.enabled)
if [ "$AUTO" != "true" ] && pgrep -f DFSZKFailoverController >/dev/null; then
  echo "ZKFC running with auto failover disabled - stop it or enable the flag"; exit 1
fi

Try / catch

try {
  nn.checkHaStateChange(req);
} catch (AccessControlException e) {
  // ZKFC-sourced request on a non-auto-HA NameNode - reconcile config, then re-send
}

Prevention

When disabling auto failover, stop ZKFC daemons in the same change
Manage ZKFC and NameNode config from one shared hdfs-site.xml source
After any HA config change, run `hdfs haadmin -failover` in a maintenance window as a smoke test

When it happens

Trigger: `hdfs haadmin -failover` (the graceful path goes through the ZKFC, which sends REQUEST_BY_ZKFC transitionToStandby to the active) or any ZKFC-initiated failover/fencing, while the target NameNode's hdfs-site.xml has dfs.ha.automatic-failover.enabled=false.

Common situations: Auto failover was disabled but the ZKFC daemons were left running; a rollback of an auto-HA rollout with stale ZKFC processes; hdfs-site.xml drift where the ZKFC host config differs from the NameNode's config.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0da536b3f0980172. Report an issue: GitHub.