apache/hadoop · warning · RetriableException

Zero blocklocations for {}. Name node is in safe mode. {} Na

Error message

Zero blocklocations for {}. Name node is in safe mode.
{} NamenodeHostName:{}

What it means

While the NameNode is in safe mode it has not yet aggregated all DataNode block reports, so blocks of recently written files may resolve to zero replica locations. getBlockLocations checks every LocatedBlock while isInSafeMode(); any block with null/empty locations triggers SafeModeException('Zero blocklocations for <src>'). In HA ACTIVE or OBSERVER state the exception is wrapped in RetriableException so DFSClient failover/retry logic re-issues the read once safe mode exits.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:2263

    FSPermissionChecker.setOperationType(operationName);
    final INode inode;
    try {
      readLock(RwLockMode.GLOBAL);
      try {
        checkOperation(OperationCategory.READ);
        res = FSDirStatAndListingOp.getBlockLocations(
            dir, pc, srcArg, offset, length, true);
        inode = res.getIIp().getLastINode();
        if (isInSafeMode()) {
          for (LocatedBlock b : res.blocks.getLocatedBlocks()) {
            // if safemode & no block locations yet then throw safemodeException
            if ((b.getLocations() == null) || (b.getLocations().length == 0)) {
              SafeModeException se = newSafemodeException(
                  "Zero blocklocations for " + srcArg);
              if (haEnabled && haContext != null &&
                  (haContext.getState().getServiceState() == ACTIVE ||
                      haContext.getState().getServiceState() == OBSERVER)) {
                throw new RetriableException(se);
              } else {
                throw se;
              }
            }
          }
        } else if (isObserver()) {
          checkBlockLocationsWhenObserver(res.blocks, srcArg);
        }
      } finally {
        readUnlock(RwLockMode.GLOBAL, operationName, getLockReportInfoSupplier(srcArg));
      }
    } catch (AccessControlException e) {
      logAuditEvent(false, operationName, srcArg);
      throw e;
    }

    logAuditEvent(true, operationName, srcArg);

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for safe mode to exit (`hdfs dfsadmin -safemode wait`) and retry the read
2. If safe mode never exits, check `hdfs dfsadmin -report` for DataNode registration and `hdfs dfsadmin -safemode get` — resolve the DataNode/block-report issue rather than forcing
3. For HA, configure the client failover proxy (dfs.client.failover.proxy.<nameservice>) so RetriableException is retried automatically
4. Only force `hdfs dfsadmin -safemode leave` when block reports are complete and you accept the risk

Example fix

// before
FSDataInputStream in = dfs.open(path); // SafeModeException / RetriableException
// after
((DistributedFileSystem) dfs).setSafeMode(HdfsConstants.SafeModeAction.WAIT); // blocks until NN leaves safe mode
FSDataInputStream in = dfs.open(path);

boolean inSafeMode = ((DistributedFileSystem) fs).setSafeMode(HdfsConstants.SafeModeAction.GET);
if (inSafeMode) {
  ((DistributedFileSystem) fs).setSafeMode(HdfsConstants.SafeModeAction.WAIT); // blocks until exit
}

catch (SafeModeException | RetriableException e) { backoffAndRetry(open); } // safe mode is transient; RetriableException exists so HA clients retry

When it happens

Trigger: Opening/reading a file (DFSClient.open -> getBlockLocations) while the NameNode is in safe mode and at least one block in the returned batch has no reported replica — typical right after NN restart or failover, or while a manual `dfsadmin -safemode enter` is active.

Common situations: Jobs start immediately after cluster restart before safe mode auto-exits; DataNodes slow or dead so the extension/missing-block thresholds keep the NN in safe mode; Observer NN serving reads right after failover with stale block maps.

Related errors

• ProcessReport from dead or unregistered node: {nodeID}
• {}. Name node is in safe mode. {} NamenodeHostName:{}
• Safe mode should be turned ON in order to create namespace i
• Unknown nameservice: {}
• Configuration has multiple addresses that match local node's

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f3c3da90c91d209a. Report an issue: GitHub.