apache/hadoop · error · IOException

Incompatible namespaceID for journal {}: NameNode has nsId {

Error message

Incompatible namespaceID for journal {}: NameNode has nsId {} but storage has nsId {}

What it means

During newEpoch (writer failover/fencing handshake), the JournalNode compares the NamespaceInfo the NameNode presents against its own stored identity. A namespaceID mismatch means this JN's storage belongs to a different namespace — typically the NN was reformatted but the JournalNodes were not, or the config points at a JN serving another namespace.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java:269

    layoutVersion = lv;
  }

  void analyzeAndRecoverStorage(StartupOption startOpt) throws IOException {
    this.state = sd.analyzeStorage(startOpt, this);
    final boolean needRecover = state != StorageState.NORMAL
        && state != StorageState.NON_EXISTENT
        && state != StorageState.NOT_FORMATTED;
    if (state == StorageState.NORMAL && startOpt != StartupOption.ROLLBACK) {
      readProperties(sd);
    } else if (needRecover) {
      sd.doRecover(state);
    }
  }

  void checkConsistentNamespace(NamespaceInfo nsInfo)
      throws IOException {
    if (nsInfo.getNamespaceID() != getNamespaceID()) {
      throw new IOException("Incompatible namespaceID for journal " +
          this.sd + ": NameNode has nsId " + nsInfo.getNamespaceID() +
          " but storage has nsId " + getNamespaceID());
    }
    
    if (!nsInfo.getClusterID().equals(getClusterID())) {
      throw new IOException("Incompatible clusterID for journal " +
          this.sd + ": NameNode has clusterId '" + nsInfo.getClusterID() +
          "' but storage has clusterId '" + getClusterID() + "'");
      
    }
  }

  public void close() throws IOException {
    LOG.info("Closing journal storage for {}", sd);
    unlockAll();
  }

  public boolean isFormatted() {

View on GitHub (pinned to 2add963021)

Solutions

1. Decide intent: keep the NN's new format, or keep the JN's old namespace. They cannot be mixed.
2. If reformatting was intentional: stop NN and JNs, clear each JN's journal dir for this journal id, start the JNs, then run 'hdfs namenode -initializeSharedEdits' from the NN to lay down matching state.
3. Verify dfs.namenome.shared.edits.dir (and each JN's dfs.journalnode.edits.dir) — a wrong hostname or port hitting another cluster's JN produces exactly this mismatch.
4. Confirm with the VERSION files: namespaceID in the NN's current/VERSION must equal the value in every JN's current/VERSION.

Example fix

# before: NN reformatted, stale JN dirs cause the mismatch
# after: reset JNs to match the freshly formatted NN
stop journalnodes & namenode
rm -rf /dfs/journal/myjournal/mynameservice/current/*
start journalnodes
hdfs --daemon start journalnode   # on each JN host
hdfs namenode -initializeSharedEdits

// Preflight before newEpoch / NN start: namespaceID must match on all sides
long nnNsId = NameNode.getNamespaceInfo().getNamespaceID();
for (URI jn : sharedEditsUris) {
  long jnNsId = readJnVersionFile(jn).namespaceID; // current/VERSION
  if (jnNsId != nnNsId) throw new IllegalStateException(
      "JN " + jn + " nsId " + jnNsId + " != NN " + nnNsId);
}

static boolean isNamespaceIdMismatch(IOException ioe) {
  return ioe.getMessage() != null && ioe.getMessage().startsWith("Incompatible namespaceID");
}

try {
  journalRpcServerCall(); // e.g. newEpoch
} catch (IOException ioe) {
  if (isNamespaceIdMismatch(ioe)) {
    // config/format problem — NEVER retry; reconcile identity first
    haltAndAlertAdmin(ioe);
  } else {
    throw ioe;
  }
}

When it happens

Trigger: NameNode calls newEpoch on a JournalNode whose dir was formatted under a different namespaceID: reformatting the NN ('hdfs namenode -format') without cleaning/re-initializing the JNs, pointing dfs.namenode.shared.edits.dir at JNs used by another nameservice, or reusing stale JN dirs.

Common situations: Test/redeploy loops that reformat the NN but keep old JN data dirs; adding the wrong JN address/port to shared edits config; reformatting after a clusterID change without touching JNs; copying a JN dir from a different cluster.

Related errors

• Incompatible clusterID for journal {}: NameNode has clusterI
• Journal {} not formatted ; journal id: {}
• ACL '{}' not of expected form scheme:id
• Parsing {} :{}
• Unknown nameservice: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d9e766cbbfe68ee. Report an issue: GitHub.