apache/hadoop · warning · IOException

DN shut down before block pool registered

Error message

DN shut down before block pool registered

What it means

BPServiceActor.registerWithNN loops sending register() until it succeeds or the actor is stopped. If the DataNode shuts down before registration completes, bpRegistration is still null when the loop exits and this IOException is thrown — the controlled exit for a DN stopped between handshake and registration.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:866

        newBpRegistration.setNamespaceInfo(nsInfo);
        bpRegistration = newBpRegistration;
        break;
      } catch(EOFException e) {  // namenode might have just restarted
        LOG.info("Problem connecting to server: {} : {}.", nnAddr, e.getLocalizedMessage());
      } catch(SocketTimeoutException e) {  // namenode is busy
        LOG.info("Problem connecting to server: {}.", nnAddr);
      } catch(RemoteException e) {
        LOG.warn("RemoteException in register to server: {}.", nnAddr, e);
        throw e;
      } catch(IOException e) {
        LOG.warn("Problem connecting to server: {}.", nnAddr);
      }
      // Try again in a second
      sleepAndLogInterrupts(1000, "connecting to server");
    }

    if (bpRegistration == null) {
      throw new IOException("DN shut down before block pool registered");
    }

    LOG.info("{} successfully registered with NN: {}.", this, nnAddr);
    bpos.registrationSucceeded(this, bpRegistration);

    // reset lease id whenever registered to NN.
    // ask for a new lease id at the next heartbeat.
    fullBlockReportLeaseId = 0;

    // random short delay - helps scatter the BR from all DNs
    scheduler.scheduleBlockReport(dnConf.initialBlockReportDelayMs, true);
  }


  private void sleepAndLogInterrupts(int millis,
      String stateString) {
    try {
      Thread.sleep(millis);

View on GitHub (pinned to 2add963021)

Solutions

1. Check DataNode logs for the shutdown trigger — this exception is the expected result of that stop, not an independent fault
2. If registration itself was looping, look for the earlier WARN 'Problem connecting to server' / 'RemoteException in register' lines for the real NN-side rejection
3. On restart, ensure the NN (and its service RPC port) is up first, then start the DN

try {
  actor.registerWithNN();
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("DN shut down before block pool registered")) {
    // expected on intentional stop; on unexpected stop, restart DN once NN is reachable
    return restartWhenNnReachable();
  }
  throw e;
}

When it happens

Trigger: DN shutdown, test teardown, or process kill arriving while the registration loop is still retrying against an unavailable NameNode (NN down, RemoteException storm, network partition).

Common situations: Operator stops a DN during an NN outage or failover; CI tears down a cluster mid-registration; orchestrator health-check timeouts kill the DN before the NN is ready.

Related errors

• DN shut down before block pool connected
• Received unimplemented DNA_SHUTDOWN
• Shutdown already in progress.
• Not ready to serve the block pool, {}.
• Cannot finalize block: {b} from Interrupted Thread

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2037c3295f06a09c. Report an issue: GitHub.