{"record":{"id":"3d9e766cbbfe68ee","repo":"apache/hadoop","slug":"incompatible-namespaceid-for-journal-namenode","errorCode":null,"errorMessage":"Incompatible namespaceID for journal {}: NameNode has nsId {} but storage has nsId {}","messagePattern":"Incompatible namespaceID for journal (.+?): NameNode has nsId (.+?) but storage has nsId (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java","lineNumber":269,"sourceCode":"    layoutVersion = lv;\n  }\n\n  void analyzeAndRecoverStorage(StartupOption startOpt) throws IOException {\n    this.state = sd.analyzeStorage(startOpt, this);\n    final boolean needRecover = state != StorageState.NORMAL\n        && state != StorageState.NON_EXISTENT\n        && state != StorageState.NOT_FORMATTED;\n    if (state == StorageState.NORMAL && startOpt != StartupOption.ROLLBACK) {\n      readProperties(sd);\n    } else if (needRecover) {\n      sd.doRecover(state);\n    }\n  }\n\n  void checkConsistentNamespace(NamespaceInfo nsInfo)\n      throws IOException {\n    if (nsInfo.getNamespaceID() != getNamespaceID()) {\n      throw new IOException(\"Incompatible namespaceID for journal \" +\n          this.sd + \": NameNode has nsId \" + nsInfo.getNamespaceID() +\n          \" but storage has nsId \" + getNamespaceID());\n    }\n    \n    if (!nsInfo.getClusterID().equals(getClusterID())) {\n      throw new IOException(\"Incompatible clusterID for journal \" +\n          this.sd + \": NameNode has clusterId '\" + nsInfo.getClusterID() +\n          \"' but storage has clusterId '\" + getClusterID() + \"'\");\n      \n    }\n  }\n\n  public void close() throws IOException {\n    LOG.info(\"Closing journal storage for {}\", sd);\n    unlockAll();\n  }\n\n  public boolean isFormatted() {","sourceCodeStart":251,"sourceCodeEnd":287,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java#L251-L287","documentation":"During newEpoch (writer failover/fencing handshake), the JournalNode compares the NamespaceInfo the NameNode presents against its own stored identity. A namespaceID mismatch means this JN's storage belongs to a different namespace — typically the NN was reformatted but the JournalNodes were not, or the config points at a JN serving another namespace.","triggerScenarios":"NameNode calls newEpoch on a JournalNode whose dir was formatted under a different namespaceID: reformatting the NN ('hdfs namenode -format') without cleaning/re-initializing the JNs, pointing dfs.namenode.shared.edits.dir at JNs used by another nameservice, or reusing stale JN dirs.","commonSituations":"Test/redeploy loops that reformat the NN but keep old JN data dirs; adding the wrong JN address/port to shared edits config; reformatting after a clusterID change without touching JNs; copying a JN dir from a different cluster.","solutions":["Decide intent: keep the NN's new format, or keep the JN's old namespace. They cannot be mixed.","If reformatting was intentional: stop NN and JNs, clear each JN's journal dir for this journal id, start the JNs, then run 'hdfs namenode -initializeSharedEdits' from the NN to lay down matching state.","Verify dfs.namenome.shared.edits.dir (and each JN's dfs.journalnode.edits.dir) — a wrong hostname or port hitting another cluster's JN produces exactly this mismatch.","Confirm with the VERSION files: namespaceID in the NN's current/VERSION must equal the value in every JN's current/VERSION."],"exampleFix":"# before: NN reformatted, stale JN dirs cause the mismatch\n# after: reset JNs to match the freshly formatted NN\nstop journalnodes & namenode\nrm -rf /dfs/journal/myjournal/mynameservice/current/*\nstart journalnodes\nhdfs --daemon start journalnode   # on each JN host\nhdfs namenode -initializeSharedEdits","handlingStrategy":"validation","validationCode":"// Preflight before newEpoch / NN start: namespaceID must match on all sides\nlong nnNsId = NameNode.getNamespaceInfo().getNamespaceID();\nfor (URI jn : sharedEditsUris) {\n  long jnNsId = readJnVersionFile(jn).namespaceID; // current/VERSION\n  if (jnNsId != nnNsId) throw new IllegalStateException(\n      \"JN \" + jn + \" nsId \" + jnNsId + \" != NN \" + nnNsId);\n}","typeGuard":"static boolean isNamespaceIdMismatch(IOException ioe) {\n  return ioe.getMessage() != null && ioe.getMessage().startsWith(\"Incompatible namespaceID\");\n}","tryCatchPattern":"try {\n  journalRpcServerCall(); // e.g. newEpoch\n} catch (IOException ioe) {\n  if (isNamespaceIdMismatch(ioe)) {\n    // config/format problem — NEVER retry; reconcile identity first\n    haltAndAlertAdmin(ioe);\n  } else {\n    throw ioe;\n  }\n}","preventionTips":["Make 'hdfs namenode -format' a documented procedure that always includes wiping and re-initializing JN dirs ('hdfs namenode -initializeSharedEdits').","Pin namespaceID explicitly (or script its propagation) in multi-node test/dev setups.","Double-check dfs.namenode.shared.edits.dir hostnames/ports against the cluster they actually serve."],"tags":["qjm","journalnode","namespace-mismatch","format","configuration"],"backgroundTag":"cluster-identity-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}