apache/hadoop · error · IOException

Unexpected BlockPoolId <blockPoolId> - expected <bpId>

Error message

Unexpected BlockPoolId <blockPoolId> - expected <bpId>

What it means

FSDirWriteFileOp.checkBlock verifies that an ExtendedBlock a client presents (for example to completeFile or getAdditionalBlock) carries the same block pool id as the NameNode namespace it is talking to. A mismatch means the client is presenting a block belonging to a different HDFS namespace, so the NameNode refuses the operation rather than update state for a foreign block.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java:148

    Block localBlock = ExtendedBlock.getLocalBlock(b);
    fsd.writeLock();
    try {
      // Remove the block from the pending creates list
      if (!unprotectedRemoveBlock(fsd, src, iip, file, localBlock)) {
        return;
      }
    } finally {
      fsd.writeUnlock();
    }
    persistBlocks(fsd, src, file, false);
  }

  static void checkBlock(FSNamesystem fsn, ExtendedBlock block)
      throws IOException {
    String bpId = fsn.getBlockPoolId();
    if (block != null && !bpId.equals(block.getBlockPoolId())) {
      throw new IOException("Unexpected BlockPoolId " + block.getBlockPoolId()
          + " - expected " + bpId);
    }
  }

  /**
   * Part I of getAdditionalBlock().
   * Analyze the state of the file under read lock to determine if the client
   * can add a new block, detect potential retries, lease mismatches,
   * and minimal replication of the penultimate block.
   *
   * Generate target DataNode locations for the new block,
   * but do not create the new block yet.
   */
  static ValidateAddBlockResult validateAddBlock(
      FSNamesystem fsn, FSPermissionChecker pc,
      String src, long fileId, String clientName,
      ExtendedBlock previous, LocatedBlock[] onRetryBlock) throws IOException {
    final long blockSize;

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the client's fs.defaultFS and ViewFS mount tables point at the nameservice that owns the block; compare the blockpoolid in the NameNode VERSION file with the block's BPID
2. If an HA NameNode has a divergent BPID, re-bootstrap or reformat the stray NameNode from the shared edits so both nodes share one namespace
3. After restores or migrations, restart long-lived clients and jobs so block references and configuration agree

Example fix

<!-- before: client writing to the wrong subcluster -->
<property><name>fs.defaultFS</name><value>hdfs://nsA</value></property>

<!-- after: route to the nameservice owning the block pool -->
<property><name>fs.defaultFS</name><value>hdfs://nsB</value></property>

URI expected = URI.create("hdfs://nsB");
if (!fs.getUri().equals(expected)) {
  throw new IllegalStateException(
      "Connected to " + fs.getUri() + " but blocks belong to " + expected);
}

try {
  namenode.complete(src, clientName, last, fileId);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Unexpected BlockPoolId")) {
    // stop retrying: this client is wired to the wrong namespace
    throw new IllegalStateException("Wrong nameservice for this block pool", e);
  }
  throw e;
}

When it happens

Trigger: completeFile/addBlock RPCs that include a block whose BPID differs from the target NameNode's BPID: client built from the wrong fs.defaultFS, ViewFS mount table routing to the wrong nameservice, or a client holding blocks from cluster A while talking to cluster B.

Common situations: Cluster migration or restore from backup producing a new block pool id while old clients keep running; HA pair where one NameNode was reformatted and diverged from shared edits; federated/ViewFS misroutes between subclusters.

Related errors

• No block pool offer service for bpid={}
• cannot locate OfferService thread for bp={blockPoolId}
• Unable to determine the name service ID. This is an HA confi
• map={} is negative. Please check {}
• {} does not support method msync

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/41e1510d9d7b9e4d. Report an issue: GitHub.