apache/hadoop · error · ServiceFailedException

Cannot transition from 'observer' to 'active'

Error message

Cannot transition from 'observer' to 'active'

What it means

transitionToActive explicitly rejects promotion from OBSERVER_STATE: the HA state machine only allows standby (or initializing) NameNodes to become active, because an Observer serves reads against a tailing namespace and lacks the edit-log ownership an active NN must assume. The transition must go through standby.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:2018

    }
    if (!getNamesystem().nameNodeHasResourcesAvailable()) {
      throw new HealthCheckFailedException(
          "The NameNode has no resources available");
    }
    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new HealthCheckFailedException("The NameNode is configured to " +
          "report UNHEALTHY to ZKFC in Safemode.");
    }
  }
  
  synchronized void transitionToActive() throws IOException {
    String operationName = "transitionToActive";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      throw new ServiceFailedException("HA for namenode is not enabled");
    }
    if (state == OBSERVER_STATE) {
      throw new ServiceFailedException(
          "Cannot transition from '" + OBSERVER_STATE + "' to '" +
              ACTIVE_STATE + "'");
    }
    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new ServiceFailedException(getRole() + " still not leave safemode");
    }
    state.setState(haContext, ACTIVE_STATE);
  }

  synchronized void transitionToStandby() throws IOException {
    String operationName = "transitionToStandby";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      throw new ServiceFailedException("HA for namenode is not enabled");
    }
    state.setState(haContext, STANDBY_STATE);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Transition observer -> standby first, then standby -> active: 'hdfs haadmin -transitionToStandby <nnId>' followed by '-transitionToActive <nnId>'.
2. Prefer promoting a genuine standby NN (one tailing edits) rather than the observer.
3. In ZKFC-managed deployments, let automatic failover handle promotion from standby instead of manual transitions.

Example fix

# before
hdfs haadmin -transitionToActive nn1   # nn1 is an Observer -> rejected

# after
hdfs haadmin -transitionToStandby nn1
hdfs haadmin -transitionToActive nn1   # observer -> standby -> active

// refuse to promote an observer directly
HAServiceStatus st = haProxy.getServiceStatus(nnId);
if ("OBSERVER".equals(String.valueOf(st.getState())))
  throw new IllegalStateException("transition observer to standby first");
haProxy.transitionToActive(nnId);

boolean isObserverToActiveDenied(Throwable t) {
  return t instanceof ServiceFailedException
      && String.valueOf(t.getMessage()).contains("'observer' to 'active'");
}

When it happens

Trigger: 'hdfs haadmin -transitionToActive <nnId>' (or HAServiceProtocol.transitionToActive) while the target NameNode's state is OBSERVER - e.g., after deploying Observer NameNodes (read-only scalable reads, Router-based federation setups).

Common situations: Operators treating observer like standby during manual failover drills; automation scripts that pick 'any non-active NN' and pick the observer; misunderstanding of the observer state model introduced for read scaling.

Related errors

• Cannot transition from 'active' to 'observer'
• {getRole()} still not leave safemode
• Failed to start active services
• {} is in observer state. Cannot be failover target
• Block pool {bpid} has not recognized an active NN

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e797a2b6d244543d. Report an issue: GitHub.