apache/hadoop · warning · RetriableException

{}. Name node is in safe mode. {} NamenodeHostName:{}

Error message

{}. Name node is in safe mode.
{} NamenodeHostName:{}

What it means

checkNameNodeSafeMode guards every namespace-mutating operation. While the NameNode is in (startup) safe mode it rejects writes with SafeModeException, appending the current safe-mode tip and host name to the caller's message. When HA is enabled, the NN is active, and it is still in startup safe mode, the exception is wrapped in RetriableException to tell clients to retry rather than fail. The condition is transient: it clears once DataNode block reports reach the safe-mode thresholds.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:1703

      getSnapshotManager().initThreadLocals();
    }
  }
  
  /**
   * @throws RetriableException
   *           If 1) The NameNode is in SafeMode, 2) HA is enabled, and 3)
   *           NameNode is in active state
   * @throws SafeModeException
   *           Otherwise if NameNode is in SafeMode.
   */
  void checkNameNodeSafeMode(String errorMsg)
      throws RetriableException, SafeModeException {
    if (isInSafeMode()) {
      SafeModeException se = newSafemodeException(errorMsg);
      if (haEnabled && haContext != null
          && haContext.getState().getServiceState() == HAServiceState.ACTIVE
          && isInStartupSafeMode()) {
        throw new RetriableException(se);
      } else {
        throw se;
      }
    }
  }

  private SafeModeException newSafemodeException(String errorMsg) {
    return new SafeModeException(errorMsg + ". Name node is in safe " +
        "mode.\n" + getSafeModeTip() + " NamenodeHostName:" + nameNodeHostName);
  }

  boolean isPermissionEnabled() {
    return isPermissionEnabled;
  }

  public static Collection<URI> getNamespaceDirs(Configuration conf) {
    return getStorageDirs(conf, DFS_NAMENODE_NAME_DIR_KEY);
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for automatic exit: 'hdfs dfsadmin -safemode wait' or poll 'hdfs dfsadmin -safemode get'
2. Make clients retry: catch SafeModeException (and RetriableException) with back-off — the state self-heals
3. Fix slow/absent DataNodes first; their block reports are what ends safe mode
4. In an emergency 'hdfs dfsadmin -safemode leave' (understand blocks may be under-replicated); before tuning, review dfs.namenode.safemode.threshold-pct and dfs.namenode.safemode.extension

Example fix

// before: single-shot call fails during NN startup
fs.mkdirs(new Path("/ingest"));   // SafeModeException

// after: wait out safe mode, then write
((DistributedFileSystem) fs).setSafeMode(
    HdfsConstants.SafeModeAction.SAFEMODE_WAIT);
fs.mkdirs(new Path("/ingest"));

// client-side pre-check before writes
DistributedFileSystem dfs = (DistributedFileSystem) FileSystem.get(conf);
while (dfs.getInSafeMode()) {
  LOG.info("NameNode in safe mode, waiting for block reports");
  Thread.sleep(5_000);
}

try {
  dfs.mkdirs(path);
} catch (SafeModeException e) {
  // startup safe mode: back off and retry — state clears itself
  Thread.sleep(5_000); /* then retry */
} catch (RetriableException e) {
  // NN told us to retry (HA active + startup safe mode)
  Thread.sleep(5_000); /* then retry */
}

When it happens

Trigger: Any write or namespace op (mkdir, delete, setReplication, setPermission, cache directive ops, saveNamespace...) invoked while isInSafeMode() is true — typically during NN startup before dfs.namenode.safemode.threshold-pct of blocks is reported, or right after a failover while the new active re-enters startup safe mode.

Common situations: Clients or startup scripts racing NN boot; slow or dead DataNodes keeping thresholds unmet; safe-mode threshold/extension misconfigured; monitoring jobs writing HDFS the moment the NN process appears up.

Related errors

• Unexpected HAServiceStateProto:
• ProcessReport from dead or unregistered node: {nodeID}
• Unrecognized section {}
• Image file is not found in {}
• Edits file is not found in {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8a553334cbdb3b76. Report an issue: GitHub.