apache/hadoop · error · AccessControlException

Request from ZK failover controller at {} denied since the n

Error message

Request from ZK failover controller at {} denied since the namenode is in Observer state.

What it means

Error "Request from ZK failover controller at {} denied since the namenode is in Observer state." thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java:1869

  public synchronized void transitionToActive(StateChangeRequestInfo req) 
      throws ServiceFailedException, AccessControlException, IOException {
    checkNNStartup();
    nn.checkHaStateChange(req);
    nn.transitionToActive();
  }

  @Override // HAServiceProtocol
  public synchronized void transitionToStandby(StateChangeRequestInfo req)
      throws ServiceFailedException, AccessControlException, IOException {
    checkNNStartup();
    // This is to eliminate any race condition between manual transition of
    // namenode into Observer, and ZKFC auto failover election, when the
    // namenode has already participated in the
    // ZKFC election, before transition to Observer state as Standby Node.
    // For more details check : HDFS-14961.
    if (nn.getState().equals(NameNode.OBSERVER_STATE.toString())
        && req.getSource() == RequestSource.REQUEST_BY_ZKFC) {
      throw new AccessControlException(
          "Request from ZK failover controller at " + Server.getRemoteAddress()
              + " denied since the namenode is in Observer state.");
    }
    nn.checkHaStateChange(req);
    nn.transitionToStandby();
  }

  @Override // HAServiceProtocol
  public synchronized void transitionToObserver(StateChangeRequestInfo req)
      throws ServiceFailedException, AccessControlException, IOException {
    checkNNStartup();
    nn.checkHaStateChange(req);
    nn.transitionToObserver();
  }

  @Override // HAServiceProtocol
  public synchronized HAServiceStatus getServiceStatus() 
      throws AccessControlException, ServiceFailedException, IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Direct ZKFC fencing/transition requests to the active or standby NameNode, not the observer; observers cannot service HA transitions.

When it happens

Trigger: The ZooKeeper failover controller sends a request to a NameNode running in Observer state.

Common situations: ZKFC misconfiguration targeting an observer node, or a failover storm hitting all nodes.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1cc5cc76e8f9f2a5. Report an issue: GitHub.