apache/hadoop · error · ServiceFailedException

{getRole()} still not leave safemode

Error message

{getRole()} still not leave safemode

What it means

transitionToActive throws ServiceFailedException '<role> still not leave safemode' when dfs.ha.nn.not-become-active-in-safemode=true and the NameNode is still in safemode. The NN refuses promotion to active until it has read enough block reports to exit safemode, so an under-initialized NN never becomes responsible for writes.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:2023

    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new HealthCheckFailedException("The NameNode is configured to " +
          "report UNHEALTHY to ZKFC in Safemode.");
    }
  }
  
  synchronized void transitionToActive() throws IOException {
    String operationName = "transitionToActive";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      throw new ServiceFailedException("HA for namenode is not enabled");
    }
    if (state == OBSERVER_STATE) {
      throw new ServiceFailedException(
          "Cannot transition from '" + OBSERVER_STATE + "' to '" +
              ACTIVE_STATE + "'");
    }
    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new ServiceFailedException(getRole() + " still not leave safemode");
    }
    state.setState(haContext, ACTIVE_STATE);
  }

  synchronized void transitionToStandby() throws IOException {
    String operationName = "transitionToStandby";
    namesystem.checkSuperuserPrivilege(operationName);
    if (!haEnabled) {
      throw new ServiceFailedException("HA for namenode is not enabled");
    }
    state.setState(haContext, STANDBY_STATE);
  }

  synchronized void transitionToObserver() throws IOException {
    String operationName = "transitionToObserver";
    namesystem.checkSuperuserPrivilege(operationName);
    if (notBecomeActiveInSafemode && isInSafeMode()) {
      throw new ServiceFailedException(getRole() + " still not leave safemode");

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for safemode to exit ('hdfs dfsadmin -safemode get' returns OFF) and retry the transition.
2. If safemode is stuck, resolve the cause: datanode availability, missing blocks ('hdfs fsck /'), then retry.
3. If your operational policy allows active-in-safemode, remove dfs.ha.nn.not-become-active-in-safemode from hdfs-site.xml.

Example fix

# before
hdfs haadmin -transitionToActive nn1   # nn1 in safemode + not-become-active-in-safemode=true

# after
hdfs dfsadmin -safemode get   # wait until 'Safe mode is OFF' (fix missing blocks if stuck)
hdfs haadmin -transitionToActive nn1

// block the transition while safemode holds
if (haConf.notBecomeActiveInSafemode() && namenode.isInSafeMode())
  throw new IllegalStateException("wait for safemode exit before transitionToActive");
namenode.transitionToActive();

boolean isSafemodeTransitionDenied(Throwable t) {
  return t instanceof ServiceFailedException
      && String.valueOf(t.getMessage()).contains("still not leave safemode");
}

When it happens

Trigger: Issuing transitionToActive (manual haadmin or failover controller) to a NN that is still in safemode while notBecomeActiveInSafemode is enabled - common right after restart or when datanodes have not reported yet.

Common situations: Post-restart failover before block reports complete; safemode extended because of missing blocks; operators manually promoting during incident recovery hitting the deliberate guard.

Related errors

• Cannot transition from 'observer' to 'active'
• Cannot transition from 'active' to 'observer'
• Failed to start active services
• Block pool {bpid} has not recognized an active NN
• The NameNode is configured to report UNHEALTHY to ZKFC in Sa

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e8e234819a348230. Report an issue: GitHub.