apache/hadoop · critical · InconsistentFSStateException

namespaceID is incompatible with others.

Error message

namespaceID is incompatible with others.

What it means

StorageInfo.setNamespaceID compares the namespaceID in the directory's VERSION with the namespaceID already loaded: if both are non-zero and differ, the directories belong to different namespaces and InconsistentFSStateException is thrown. An empty namespaceID is not permitted here (getProperty would already have failed on a missing key), so the mismatch is always a real conflict.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/StorageInfo.java:231

  }
  
  /** Validate and set layout version from {@link Properties}*/
  protected void setLayoutVersion(Properties props, StorageDirectory sd)
      throws IncorrectVersionException, InconsistentFSStateException {
    int lv = Integer.parseInt(getProperty(props, sd, "layoutVersion"));
    if (lv < getServiceLayoutVersion()) { // future version
      throw new IncorrectVersionException(getServiceLayoutVersion(), lv,
          "storage directory " + sd.root.getAbsolutePath());
    }
    layoutVersion = lv;
  }
  
  /** Validate and set namespaceID version from {@link Properties}*/
  protected void setNamespaceID(Properties props, StorageDirectory sd)
      throws InconsistentFSStateException {
    int nsId = Integer.parseInt(getProperty(props, sd, "namespaceID"));
    if (namespaceID != 0 && nsId != 0 && namespaceID != nsId) {
      throw new InconsistentFSStateException(sd.root,
          "namespaceID is incompatible with others.");
    }
    namespaceID = nsId;
  }

  public void setServiceLayoutVersion(int lv) {
    this.layoutVersion = lv;
  }

  public int getServiceLayoutVersion() {
    return storageType == NodeType.DATA_NODE
        ? DataNodeLayoutVersion.getCurrentLayoutVersion()
        : HdfsServerConstants.NAMENODE_LAYOUT_VERSION;
  }

  public Map<Integer, SortedSet<LayoutFeature>> getServiceLayoutFeatureMap() {
    return storageType == NodeType.DATA_NODE? DataNodeLayoutVersion.FEATURES
        : NameNodeLayoutVersion.FEATURES;

View on GitHub (pinned to 2add963021)

Solutions

  1. Compare namespaceID across <dir>/VERSION files of all configured directories to find the mismatched one
  2. Remove the mismatched directory from the configuration, or wipe/reformat it so it adopts the running namespace
  3. After reformatting a NameNode, also clear DataNode storage directories so namespaceIDs realign on re-registration
Defensive patterns

Strategy: validation

Validate before calling

Set<Integer> ids = new HashSet<>();
for (String d : configuredDirs) {
  Properties p = Storage.readPropertiesFile(new File(d, "VERSION"));
  int nsId = Integer.parseInt(p.getProperty("namespaceID"));
  if (nsId != 0) ids.add(nsId);
}
if (ids.size() > 1) throw new IOException("mixed namespaceIDs in storage dirs: " + ids);

Prevention

When it happens

Trigger: Mixing a freshly formatted storage directory (new namespaceID) with directories of an existing namespace; a DataNode dir from a different namespace left in the config; NameNode was reformatted but DataNode dirs were not cleared.

Common situations: Adding a newly formatted volume to an existing set of storage dirs; reformatting the NameNode in a test loop without wiping DN storage; restoring one dir from another cluster's backup.

Related errors


AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/18ef63437876f52a. Report an issue: GitHub.