apache/hadoop · error · IOException

cannot locate OfferService thread for bp={blockPoolId}

Error message

cannot locate OfferService thread for bp={blockPoolId}

What it means

getBPOSForBlock resolves the BPOfferService (the thread serving one block pool / NameNode namespace) for an ExtendedBlock's blockPoolId via blockPoolManager. If no BPOS is registered for that id, the DN throws IOException("cannot locate OfferService thread for bp=...") — the block belongs to a namespace this DataNode does not serve, does not yet serve, or has just stopped serving.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1838

          corruptionMap.entrySet()) {
        for (DatanodeInfo dnInfo : entry.getValue()) {
          reportRemoteBadBlock(dnInfo, entry.getKey());
        }
      }
    }
  }

  /**
   * Return the BPOfferService instance corresponding to the given block.
   * @return the BPOS
   * @throws IOException if no such BPOS can be found
   */
  private BPOfferService getBPOSForBlock(ExtendedBlock block)
      throws IOException {
    Preconditions.checkNotNull(block);
    BPOfferService bpos = blockPoolManager.get(block.getBlockPoolId());
    if (bpos == null) {
      throw new IOException("cannot locate OfferService thread for bp="+
          block.getBlockPoolId());
    }
    return bpos;
  }

  // used only for testing
  @VisibleForTesting
  public void setHeartbeatsDisabledForTests(
      boolean heartbeatsDisabledForTests) {
    this.heartbeatsDisabledForTests = heartbeatsDisabledForTests;
  }

  @VisibleForTesting
  boolean areHeartbeatsDisabledForTests() {
    return this.heartbeatsDisabledForTests;
  }

  @VisibleForTesting

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm which block pools this DN serves (DN logs 'Block pool bp-... registered', or JMX) and compare with the bpid in the error
2. If a namespace is missing on the DN, fix dfs.nameservices config on the NN side and run hdfs dfsadmin -refreshNamenodes <dn:ipcPort> so the DN (re)registers
3. In tests/tools, stop referencing blocks from torn-down namespaces; fetch blocks via the live BPOS

if (block == null || block.getBlockPoolId() == null
    || blockPoolManager.get(block.getBlockPoolId()) == null) {
  LOG.warn("Unknown block pool {}", block.getBlockPoolId());
  return; // do not call getBPOSForBlock
}

try {
  BPOfferService bpos = getBPOSForBlock(block);
} catch (IOException e) {
  if (e.getMessage().contains("cannot locate OfferService thread")) {
    // block pool not served here: drop the request and log the bpid; check refreshNamenames/federation config
  } else { throw e; }
}

When it happens

Trigger: Handling a block whose bpid comes from a different cluster or federation namespace (miswired dfs.nameservices, cross-cluster tooling replaying requests); DN still starting before BPOS registration finishes; refreshNamenodes just removed that namespace while a request was in flight; tests holding stale ExtendedBlock objects after a namespace was torn down.

Common situations: Federation misconfiguration; HA failover races during DN re-registration; unit tests reusing blocks after stopping a block pool; scripts pointing at the wrong cluster.

Related errors

• No block pool offer service for bpid={}
• Storage directory for location {} and block pool id {} does
• Not a valid Boolean value for {property} in reconfSlowPeerPa
• FsDatasetSpi has not been initialized
• Not a valid Boolean value for {property}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a049305497dc3ff0. Report an issue: GitHub.