apache/hadoop · error · IOException

No block pool offer service for bpid={}

Error message

No block pool offer service for bpid={}

What it means

BlockRecoveryWorker.getDatanodeID asks datanode.getBPOfferService(bpid) for the service actor handling that block pool. A null BPOfferService means this datanode is not (or no longer) serving that block pool id — it has no registration, namespace info, or NN connection for it — so recovery work for that pool cannot proceed and an IOException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:565

        ReplicaRecoveryInfo rInfo = r.getReplicaRecoveryInfo();
        blockLengths[i++] = rInfo.getNumBytes();
      }
      return StripedBlockUtil.getSafeLength(ecPolicy, blockLengths);
    }

    private void checkLocations(int locationCount)
        throws IOException {
      if (locationCount < ecPolicy.getNumDataUnits()) {
        throw new IOException(block + " has no enough internal blocks(current: " + locationCount +
            "), unable to start recovery. Locations=" + Arrays.asList(locs));
      }
    }
  }

  private DatanodeID getDatanodeID(String bpid) throws IOException {
    BPOfferService bpos = datanode.getBPOfferService(bpid);
    if (bpos == null) {
      throw new IOException("No block pool offer service for bpid=" + bpid);
    }
    return new DatanodeID(bpos.bpRegistration);
  }

  private static void logRecoverBlock(String who, RecoveringBlock rb) {
    ExtendedBlock block = rb.getBlock();
    DatanodeInfo[] targets = rb.getLocations();

    LOG.info("BlockRecoveryWorker: {} calls recoverBlock({}, targets=[{}], newGenerationStamp={}"
        + ", newBlock={}, isStriped={})", who, block, Joiner.on(", ").join(targets),
        rb.getNewGenerationStamp(), rb.getNewBlock(), rb.isStriped());
  }

  /**
   * Convenience method, which unwraps RemoteException.
   * @throws IOException not a RemoteException.
   */
  private static ReplicaRecoveryInfo callInitReplicaRecovery(

View on GitHub (pinned to 2add963021)

Solutions

1. Check the DN's configured nameservices (dfs.nameservices, dfs.namenode.rpc-address per nameservice) include the namespace owning the bpid.
2. Confirm the datanode has finished startup and successfully handshake-registered with every NameNode (look for 'Successfully sent block report' / registration logs per BPOfferService).
3. If the command came from a decommissioned/renamed namespace, refresh the NN-side layout or discard stale recovery requests.
4. Restart the DataNode after fixing configuration so BPOfferServices are (re)created for all block pools.

// Confirm this DN actually serves the block pool before recovery work
if (datanode.getBPOfferService(extendedBlock.getBlockPoolId()) == null) {
  LOG.warn("DN does not serve bpid={}, skipping recovery command", extendedBlock.getBlockPoolId());
  return;
}

try {
  DatanodeID id = worker.getDatanodeID(bpid);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("No block pool offer service")) {
    // configuration/registration gap: surface config error, do not blind-retry
    throw new IllegalStateException("DN misconfigured for block pool " + bpid, e);
  }
  throw e;
}

When it happens

Trigger: A recovery command arrives carrying a bpid for which this DataNode has no BPOfferService: dfs.nameservices / dfs.internal.nameseries federation config on the DN does not include that namespace, the DN is mid-startup before handshake, or it is shutting down and the actor was already removed.

Common situations: Federation misconfiguration where the datanode's dfs.nameservices list omits one nameservice; a stale or misrouted NN command; races during DataNode restart or rolling upgrade while recovery commands are still in flight.

Related errors

• cannot locate OfferService thread for bp={blockPoolId}
• Unknown nameservice: {}
• Incompatible node types: storageType={storageType} but Stora
• No services to connect, missing NameNode address.
• Storage directory for location {} and block pool id {} does

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/39dcf5a832f658d8. Report an issue: GitHub.