apache/hadoop · error · IOException

Block pool {bpid} has not recognized an active NN

Error message

Block pool {bpid} has not recognized an active NN

What it means

getActiveNamenodeForBP found a BPOfferService for the block pool, but bpos.getActiveNN() returned null: none of the service actors for that pool has observed a transition to active. In an HA nameservice this means the DN currently sees only standby NameNodes (or has not yet processed the active's heartbeat state), so there is no DatanodeProtocol endpoint to commit the recovery result to and the IOException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:609

  }

  /**
   * Get the NameNode corresponding to the given block pool.
   *
   * @param bpid Block pool Id
   * @return Namenode corresponding to the bpid
   * @throws IOException if unable to get the corresponding NameNode
   */
  DatanodeProtocolClientSideTranslatorPB getActiveNamenodeForBP(
      String bpid) throws IOException {
    BPOfferService bpos = datanode.getBPOfferService(bpid);
    if (bpos == null) {
      throw new IOException("No block pool offer service for bpid=" + bpid);
    }

    DatanodeProtocolClientSideTranslatorPB activeNN = bpos.getActiveNN();
    if (activeNN == null) {
      throw new IOException(
          "Block pool " + bpid + " has not recognized an active NN");
    }
    return activeNN;
  }

  public Daemon recoverBlocks(final String who,
      final Collection<RecoveringBlock> blocks) {
    Daemon d = new Daemon(datanode.threadGroup, new Runnable() {
      @Override
      public void run() {
        datanode.metrics.incrDataNodeBlockRecoveryWorkerCount();
        try {
          for (RecoveringBlock b : blocks) {
            try {
              logRecoverBlock(who, b);
              if (b.isStriped()) {
                new RecoveryTaskStriped((RecoveringStripedBlock) b).recover();
              } else {

View on GitHub (pinned to 2add963021)

Solutions

1. Check NameNode HA state (haadmin -getServiceState / automatic failover controller logs) and ensure exactly one active NN is up and reachable.
2. If a failover was in flight, no action is needed: once the DN recognizes the new active, the NN re-requests block recovery and the commit succeeds.
3. Verify the DN can reach the active NN's RPC address (dfs.namenode.rpc-address.[ns].[nn]) and check for actor connection errors in the DN log.
4. For repeated occurrences, inspect ZKFC/quorum health so a nameservice never sits without a recognized active.

// Before committing recovery, verify the pool has a recognized active NN
BPOfferService bpos = datanode.getBPOfferService(bpid);
if (bpos == null || bpos.getActiveNN() == null) {
  LOG.info("No active NN recognized for bpid={} yet; deferring commit", bpid);
  return; // NN will re-request recovery once active is recognized
}

try {
  nn = worker.getActiveNamenodeForBP(bpid);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("has not recognized an active NN")) {
    retryWithBackoff(); // HA failover window — the active will be recognized shortly
  } else {
    throw e;
  }
}

When it happens

Trigger: commitBlockSynchronization during an HA failover window (old active lost, new active not yet recognized); all NameNodes of the nameservice in standby; DN freshly started and still learning the active; brief ZKFC/leader-election outages leaving no active NN.

Common situations: NameNode failover storms, a standby-only nameservice after a crash, network isolation between DN and the active NN's RPC port, or recovery threads committing right as the active NN restarts.

Related errors

• " + idHelpText + " mismatch: previously connected to " + idH
• Cannot transition from 'observer' to 'active'
• {getRole()} still not leave safemode
• Cannot transition from 'active' to 'observer'
• Failed to start active services

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/27c72cfab715ee9c. Report an issue: GitHub.