apache/hadoop · warning · IOException

DN shut down before block pool connected

Error message

DN shut down before block pool connected

What it means

During startup, BPServiceActor loops trying to retrieve NamespaceInfo from the NameNode for as long as the actor should run. If the DataNode is shut down (BPOfferService stopped) before a successful handshake completes, the loop exits with nsInfo null and this IOException is thrown instead of retrying forever. It is the controlled termination path for a DN stopped while the NN was still unreachable.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:277

    while (shouldRun()) {
      try {
        nsInfo = bpNamenode.versionRequest();
        LOG.debug("{} received versionRequest response: {}", this, nsInfo);
        break;
      } catch(SocketTimeoutException e) {  // namenode is busy
        LOG.warn("Problem connecting to server: " + nnAddr);
      } catch(IOException e ) {  // namenode is not available
        LOG.warn("Problem connecting to server: " + nnAddr);
      }
      
      // try again in a second
      sleepAndLogInterrupts(5000, "requesting version info from NN");
    }
    
    if (nsInfo != null) {
      checkNNVersion(nsInfo);
    } else {
      throw new IOException("DN shut down before block pool connected");
    }
    return nsInfo;
  }

  private void checkNNVersion(NamespaceInfo nsInfo)
      throws IncorrectVersionException {
    // build and layout versions should match
    String nnVersion = nsInfo.getSoftwareVersion();
    String minimumNameNodeVersion = dnConf.getMinimumNameNodeVersion();
    if (VersionUtil.compareVersions(nnVersion, minimumNameNodeVersion) < 0) {
      IncorrectVersionException ive = new IncorrectVersionException(
          minimumNameNodeVersion, nnVersion, "NameNode", "DataNode");
      LOG.warn(ive.getMessage());
      throw ive;
    }
    String dnVersion = VersionInfo.getVersion();
    if (!nnVersion.equals(dnVersion)) {
      LOG.info("Reported NameNode version '" + nnVersion + "' does not match " +

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm in DataNode logs that a shutdown was actually requested — this exception is the expected consequence, not the root cause
2. If the stop was unexpected, find what killed the DN (OOM killer in dmesg, orchestrator) and address that
3. On next start, ensure the NN is reachable: verify dfs.namenode.servicerpc-address / HA config and network before the DN comes up

// In DN lifecycle wrappers: an intentional stop must not surface as an error
try {
  actor.connectToNN();
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("DN shut down before block pool connected")) {
    LOG.info("DN stopped while waiting for NameNode; exiting actor cleanly");
    return; // expected shutdown path
  }
  throw e;
}

When it happens

Trigger: DN shutdown/stop command, test teardown, or container kill arriving while connectToNN is still retrying against an unreachable or slow NameNode.

Common situations: Restart scripts stopping a DN that cannot reach the NN (NN down, service-rpc address wrong, firewall); systemd/docker teardown during long NN outages; integration tests aborting before first handshake.

Related errors

• DN shut down before block pool registered
• Received unimplemented DNA_SHUTDOWN
• Shutdown already in progress.
• Cannot finalize block: {b} from Interrupted Thread
• The volume has already closed.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0be26c76aa833e20. Report an issue: GitHub.