apache/hadoop · critical · InconsistentFSStateException

Unexepcted blockpoolID " + bpid + " . Expected " + blockpool

Error message

Unexepcted blockpoolID " + bpid + " . Expected " + blockpoolID

What it means

The same validator rejects a bpid that differs from the one already loaded: blockpoolID is set from the first directory read and every storage directory of the nameservice must agree, so 'Unexepcted blockpoolID <b> . Expected <e>' (the typo is in the Hadoop source) means directories from two different formats or clusters are mixed in one NameNode.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java:1059

    int rand = DFSUtil.getSecureRandom().nextInt(Integer.MAX_VALUE);
    return "BP-" + rand + "-"+ ip + "-" + Time.now();
  }

  /** Validate and set block pool ID. */
  public void setBlockPoolID(String bpid) {
    blockpoolID = bpid;
  }

  /** Validate and set block pool ID. */
  private void setBlockPoolID(File storage, String bpid)
      throws InconsistentFSStateException {
    if (bpid == null || bpid.equals("")) {
      throw new InconsistentFSStateException(storage, "file "
          + Storage.STORAGE_FILE_VERSION + " has no block pool Id.");
    }
    
    if (!blockpoolID.equals("") && !blockpoolID.equals(bpid)) {
      throw new InconsistentFSStateException(storage,
          "Unexepcted blockpoolID " + bpid + " . Expected " + blockpoolID);
    }
    setBlockPoolID(bpid);
  }
  
  public String getBlockPoolID() {
    return blockpoolID;
  }

  /**
   * Iterate over all current storage directories, inspecting them
   * with the given inspector.
   */
  void inspectStorageDirs(FSImageStorageInspector inspector)
      throws IOException {

    // Process each of the storage directories to find the pair of
    // newest image file and edit file

View on GitHub (pinned to 2add963021)

Solutions

Compare the blockpoolID= lines across all name/edits dirs' VERSION files; the odd one out came from another namespace.
Empty the offending dir (or reformat the whole cluster if disposable) so all dirs share one BP id - never mix dirs across namespaces.
If the mismatched dir holds real data, it belongs to the other cluster: mount it there instead.

Defensive patterns

Strategy: validation

Validate before calling

// Assert a single blockpoolID across all storage dirs before NN start
Set<String> bpids = new HashSet<>();
for (String loc : conf.getTrimmedStrings("dfs.namenode.name.dir")) {
  bpids.add(parseVersion(Paths.get(stripScheme(loc), "current", "VERSION")).get("blockpoolID"));
}
if (bpids.size() != 1 || bpids.contains(null)) {
  throw new IllegalStateException("Mixed blockpoolIDs across name dirs: " + bpids
      + " - a dir from another namespace is configured");
}

Prevention

Wipe or reformat reused disks before adding them as name dirs.
During reformats, format all dirs in one operation so BP ids cannot diverge.
Diff blockpoolID= lines across dirs as part of pre-start checks.

When it happens

Trigger: One dfs.namenode.name.dir (or edits dir) belongs to another cluster/namespace - a disk reused from a different cluster, or only a subset of directories was reformatted during the last format.

Common situations: Reformatting only some dirs; attaching old data disks to a new NameNode; test clusters sharing storage across namespaces; swap-in of a dir from a twin cluster.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0d0209029c1a0d0c. Report an issue: GitHub.