{"record":{"id":"39dcf5a832f658d8","repo":"apache/hadoop","slug":"no-block-pool-offer-service-for-bpid","errorCode":null,"errorMessage":"No block pool offer service for bpid={}","messagePattern":"No block pool offer service for bpid=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java","lineNumber":565,"sourceCode":"        ReplicaRecoveryInfo rInfo = r.getReplicaRecoveryInfo();\n        blockLengths[i++] = rInfo.getNumBytes();\n      }\n      return StripedBlockUtil.getSafeLength(ecPolicy, blockLengths);\n    }\n\n    private void checkLocations(int locationCount)\n        throws IOException {\n      if (locationCount < ecPolicy.getNumDataUnits()) {\n        throw new IOException(block + \" has no enough internal blocks(current: \" + locationCount +\n            \"), unable to start recovery. Locations=\" + Arrays.asList(locs));\n      }\n    }\n  }\n\n  private DatanodeID getDatanodeID(String bpid) throws IOException {\n    BPOfferService bpos = datanode.getBPOfferService(bpid);\n    if (bpos == null) {\n      throw new IOException(\"No block pool offer service for bpid=\" + bpid);\n    }\n    return new DatanodeID(bpos.bpRegistration);\n  }\n\n  private static void logRecoverBlock(String who, RecoveringBlock rb) {\n    ExtendedBlock block = rb.getBlock();\n    DatanodeInfo[] targets = rb.getLocations();\n\n    LOG.info(\"BlockRecoveryWorker: {} calls recoverBlock({}, targets=[{}], newGenerationStamp={}\"\n        + \", newBlock={}, isStriped={})\", who, block, Joiner.on(\", \").join(targets),\n        rb.getNewGenerationStamp(), rb.getNewBlock(), rb.isStriped());\n  }\n\n  /**\n   * Convenience method, which unwraps RemoteException.\n   * @throws IOException not a RemoteException.\n   */\n  private static ReplicaRecoveryInfo callInitReplicaRecovery(","sourceCodeStart":547,"sourceCodeEnd":583,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java#L547-L583","documentation":"BlockRecoveryWorker.getDatanodeID asks datanode.getBPOfferService(bpid) for the service actor handling that block pool. A null BPOfferService means this datanode is not (or no longer) serving that block pool id — it has no registration, namespace info, or NN connection for it — so recovery work for that pool cannot proceed and an IOException is thrown.","triggerScenarios":"A recovery command arrives carrying a bpid for which this DataNode has no BPOfferService: dfs.nameservices / dfs.internal.nameseries federation config on the DN does not include that namespace, the DN is mid-startup before handshake, or it is shutting down and the actor was already removed.","commonSituations":"Federation misconfiguration where the datanode's dfs.nameservices list omits one nameservice; a stale or misrouted NN command; races during DataNode restart or rolling upgrade while recovery commands are still in flight.","solutions":["Check the DN's configured nameservices (dfs.nameservices, dfs.namenode.rpc-address per nameservice) include the namespace owning the bpid.","Confirm the datanode has finished startup and successfully handshake-registered with every NameNode (look for 'Successfully sent block report' / registration logs per BPOfferService).","If the command came from a decommissioned/renamed namespace, refresh the NN-side layout or discard stale recovery requests.","Restart the DataNode after fixing configuration so BPOfferServices are (re)created for all block pools."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Confirm this DN actually serves the block pool before recovery work\nif (datanode.getBPOfferService(extendedBlock.getBlockPoolId()) == null) {\n  LOG.warn(\"DN does not serve bpid={}, skipping recovery command\", extendedBlock.getBlockPoolId());\n  return;\n}","typeGuard":null,"tryCatchPattern":"try {\n  DatanodeID id = worker.getDatanodeID(bpid);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"No block pool offer service\")) {\n    // configuration/registration gap: surface config error, do not blind-retry\n    throw new IllegalStateException(\"DN misconfigured for block pool \" + bpid, e);\n  }\n  throw e;\n}","preventionTips":["Keep dfs.nameservices consistent across all NameNodes and DataNodes when federating.","Only dispatch recovery work after the DN reports successful registration with the owning namespace.","Restart DataNodes after any federation-level configuration change."],"tags":["hdfs","datanode","block-pool","federation","configuration"],"backgroundTag":"unknown-block-pool","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}