{"record":{"id":"0da536b3f0980172","repo":"apache/hadoop","slug":"request-from-zk-failover-controller-at-server-get","errorCode":null,"errorMessage":"Request from ZK failover controller at {Server.getRemoteAddress()} denied since automatic HA is not enabled","messagePattern":"Request from ZK failover controller at (.+?) denied since automatic HA is not enabled","errorType":"exception","errorClass":"AccessControlException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java","lineNumber":2323,"sourceCode":"    switch (req.getSource()) {\n    case REQUEST_BY_USER:\n      if (autoHaEnabled) {\n        throw new AccessControlException(\n            \"Manual HA control for this NameNode is disallowed, because \" +\n            \"automatic HA is enabled.\");\n      }\n      break;\n    case REQUEST_BY_USER_FORCED:\n      if (autoHaEnabled) {\n        LOG.warn(\"Allowing manual HA control from \" +\n            Server.getRemoteAddress() +\n            \" even though automatic HA is enabled, because the user \" +\n            \"specified the force flag\");\n      }\n      break;\n    case REQUEST_BY_ZKFC:\n      if (!autoHaEnabled) {\n        throw new AccessControlException(\n            \"Request from ZK failover controller at \" +\n            Server.getRemoteAddress() + \" denied since automatic HA \" +\n            \"is not enabled\"); \n      }\n      break;\n    }\n  }\n\n  /*\n   * {@inheritDoc}\n   * */\n  @Override // ReconfigurableBase\n  public Collection<String> getReconfigurableProperties() {\n    return reconfigurableProperties;\n  }\n\n  /*\n   * {@inheritDoc}","sourceCodeStart":2305,"sourceCodeEnd":2341,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L2305-L2341","documentation":"The REQUEST_BY_ZKFC arm of NameNode.checkHaStateChange: a state-change request sourced from the ZKFailoverController (graceful failover or fencing transitions) is rejected with AccessControlException when this NameNode's conf has dfs.ha.automatic-failover.enabled=false. The NameNode refuses ZKFC-driven transitions it was not configured to accept, and the message names the ZKFC's address via Server.getRemoteAddress(). Note the asymmetry with the REQUEST_BY_USER arm: the same boolean guards both arms in opposite directions.","triggerScenarios":"`hdfs haadmin -failover` (the graceful path goes through the ZKFC, which sends REQUEST_BY_ZKFC transitionToStandby to the active) or any ZKFC-initiated failover/fencing, while the target NameNode's hdfs-site.xml has dfs.ha.automatic-failover.enabled=false.","commonSituations":"Auto failover was disabled but the ZKFC daemons were left running; a rollback of an auto-HA rollout with stale ZKFC processes; hdfs-site.xml drift where the ZKFC host config differs from the NameNode's config.","solutions":["If automatic failover is no longer wanted, stop the ZKFC daemons (`hdfs --daemon stop zkfc`) so nothing emits REQUEST_BY_ZKFC requests","If automatic failover is wanted, set dfs.ha.automatic-failover.enabled=true in the NameNode's hdfs-site.xml and restart the NameNode","Diff hdfs-site.xml across every HA node and ZKFC host - the flag must agree cluster-wide, since the NN judges only its own copy","After fixing, verify the ZKFC path with `hdfs haadmin -failover` on a test pair"],"exampleFix":"<!-- hdfs-site.xml on the NameNode: before -->\n<property><name>dfs.ha.automatic-failover.enabled</name><value>false</value></property>\n\n<!-- after (then restart NameNode and ZKFC) -->\n<property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property>","handlingStrategy":"validation","validationCode":"# Guard: never run ZKFC while the NN disables auto failover\nAUTO=$(hdfs getconf -confKey dfs.ha.automatic-failover.enabled)\nif [ \"$AUTO\" != \"true\" ] && pgrep -f DFSZKFailoverController >/dev/null; then\n  echo \"ZKFC running with auto failover disabled - stop it or enable the flag\"; exit 1\nfi","typeGuard":null,"tryCatchPattern":"try {\n  nn.checkHaStateChange(req);\n} catch (AccessControlException e) {\n  // ZKFC-sourced request on a non-auto-HA NameNode - reconcile config, then re-send\n}","preventionTips":["When disabling auto failover, stop ZKFC daemons in the same change","Manage ZKFC and NameNode config from one shared hdfs-site.xml source","After any HA config change, run `hdfs haadmin -failover` in a maintenance window as a smoke test"],"tags":["hdfs","name-node","ha","zkfc","failover","access-control","config-drift"],"backgroundTag":"ha-failover-access-denied","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}