{"record":{"id":"2037c3295f06a09c","repo":"apache/hadoop","slug":"dn-shut-down-before-block-pool-registered","errorCode":null,"errorMessage":"DN shut down before block pool registered","messagePattern":"DN shut down before block pool registered","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java","lineNumber":866,"sourceCode":"        newBpRegistration.setNamespaceInfo(nsInfo);\n        bpRegistration = newBpRegistration;\n        break;\n      } catch(EOFException e) {  // namenode might have just restarted\n        LOG.info(\"Problem connecting to server: {} : {}.\", nnAddr, e.getLocalizedMessage());\n      } catch(SocketTimeoutException e) {  // namenode is busy\n        LOG.info(\"Problem connecting to server: {}.\", nnAddr);\n      } catch(RemoteException e) {\n        LOG.warn(\"RemoteException in register to server: {}.\", nnAddr, e);\n        throw e;\n      } catch(IOException e) {\n        LOG.warn(\"Problem connecting to server: {}.\", nnAddr);\n      }\n      // Try again in a second\n      sleepAndLogInterrupts(1000, \"connecting to server\");\n    }\n\n    if (bpRegistration == null) {\n      throw new IOException(\"DN shut down before block pool registered\");\n    }\n\n    LOG.info(\"{} successfully registered with NN: {}.\", this, nnAddr);\n    bpos.registrationSucceeded(this, bpRegistration);\n\n    // reset lease id whenever registered to NN.\n    // ask for a new lease id at the next heartbeat.\n    fullBlockReportLeaseId = 0;\n\n    // random short delay - helps scatter the BR from all DNs\n    scheduler.scheduleBlockReport(dnConf.initialBlockReportDelayMs, true);\n  }\n\n\n  private void sleepAndLogInterrupts(int millis,\n      String stateString) {\n    try {\n      Thread.sleep(millis);","sourceCodeStart":848,"sourceCodeEnd":884,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java#L848-L884","documentation":"BPServiceActor.registerWithNN loops sending register() until it succeeds or the actor is stopped. If the DataNode shuts down before registration completes, bpRegistration is still null when the loop exits and this IOException is thrown — the controlled exit for a DN stopped between handshake and registration.","triggerScenarios":"DN shutdown, test teardown, or process kill arriving while the registration loop is still retrying against an unavailable NameNode (NN down, RemoteException storm, network partition).","commonSituations":"Operator stops a DN during an NN outage or failover; CI tears down a cluster mid-registration; orchestrator health-check timeouts kill the DN before the NN is ready.","solutions":["Check DataNode logs for the shutdown trigger — this exception is the expected result of that stop, not an independent fault","If registration itself was looping, look for the earlier WARN 'Problem connecting to server' / 'RemoteException in register' lines for the real NN-side rejection","On restart, ensure the NN (and its service RPC port) is up first, then start the DN"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  actor.registerWithNN();\n} catch (IOException e) {\n  if (e.getMessage() != null\n      && e.getMessage().contains(\"DN shut down before block pool registered\")) {\n    // expected on intentional stop; on unexpected stop, restart DN once NN is reachable\n    return restartWhenNnReachable();\n  }\n  throw e;\n}","preventionTips":["Sequence cluster startup: NN ready, then DN","Give DNs enough start-up grace in orchestrators so health checks don't kill them mid-registration"],"tags":["hdfs","datanode","shutdown","registration","lifecycle"],"backgroundTag":"shutdown-race","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}