{"record":{"id":"0be26c76aa833e20","repo":"apache/hadoop","slug":"dn-shut-down-before-block-pool-connected","errorCode":null,"errorMessage":"DN shut down before block pool connected","messagePattern":"DN shut down before block pool connected","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java","lineNumber":277,"sourceCode":"    while (shouldRun()) {\n      try {\n        nsInfo = bpNamenode.versionRequest();\n        LOG.debug(\"{} received versionRequest response: {}\", this, nsInfo);\n        break;\n      } catch(SocketTimeoutException e) {  // namenode is busy\n        LOG.warn(\"Problem connecting to server: \" + nnAddr);\n      } catch(IOException e ) {  // namenode is not available\n        LOG.warn(\"Problem connecting to server: \" + nnAddr);\n      }\n      \n      // try again in a second\n      sleepAndLogInterrupts(5000, \"requesting version info from NN\");\n    }\n    \n    if (nsInfo != null) {\n      checkNNVersion(nsInfo);\n    } else {\n      throw new IOException(\"DN shut down before block pool connected\");\n    }\n    return nsInfo;\n  }\n\n  private void checkNNVersion(NamespaceInfo nsInfo)\n      throws IncorrectVersionException {\n    // build and layout versions should match\n    String nnVersion = nsInfo.getSoftwareVersion();\n    String minimumNameNodeVersion = dnConf.getMinimumNameNodeVersion();\n    if (VersionUtil.compareVersions(nnVersion, minimumNameNodeVersion) < 0) {\n      IncorrectVersionException ive = new IncorrectVersionException(\n          minimumNameNodeVersion, nnVersion, \"NameNode\", \"DataNode\");\n      LOG.warn(ive.getMessage());\n      throw ive;\n    }\n    String dnVersion = VersionInfo.getVersion();\n    if (!nnVersion.equals(dnVersion)) {\n      LOG.info(\"Reported NameNode version '\" + nnVersion + \"' does not match \" +","sourceCodeStart":259,"sourceCodeEnd":295,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java#L259-L295","documentation":"During startup, BPServiceActor loops trying to retrieve NamespaceInfo from the NameNode for as long as the actor should run. If the DataNode is shut down (BPOfferService stopped) before a successful handshake completes, the loop exits with nsInfo null and this IOException is thrown instead of retrying forever. It is the controlled termination path for a DN stopped while the NN was still unreachable.","triggerScenarios":"DN shutdown/stop command, test teardown, or container kill arriving while connectToNN is still retrying against an unreachable or slow NameNode.","commonSituations":"Restart scripts stopping a DN that cannot reach the NN (NN down, service-rpc address wrong, firewall); systemd/docker teardown during long NN outages; integration tests aborting before first handshake.","solutions":["Confirm in DataNode logs that a shutdown was actually requested — this exception is the expected consequence, not the root cause","If the stop was unexpected, find what killed the DN (OOM killer in dmesg, orchestrator) and address that","On next start, ensure the NN is reachable: verify dfs.namenode.servicerpc-address / HA config and network before the DN comes up"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// In DN lifecycle wrappers: an intentional stop must not surface as an error\ntry {\n  actor.connectToNN();\n} catch (IOException e) {\n  if (e.getMessage() != null\n      && e.getMessage().contains(\"DN shut down before block pool connected\")) {\n    LOG.info(\"DN stopped while waiting for NameNode; exiting actor cleanly\");\n    return; // expected shutdown path\n  }\n  throw e;\n}","preventionTips":["Ensure NN service RPC endpoints are up before starting DNs (start order in scripts/orchestrators)","Treat 'DN shut down before …' exceptions as symptoms of a stop, and always locate the shutdown trigger in logs"],"tags":["hdfs","datanode","shutdown","namenode-connection","lifecycle"],"backgroundTag":"shutdown-race","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}