apache/hadoop · error · IOException

Incompatible clusterID for journal {}: NameNode has clusterI

Error message

Incompatible clusterID for journal {}: NameNode has clusterId '{}' but storage has clusterId '{}'

What it means

The clusterID half of JNStorage.checkConsistentNamespace: during newEpoch the JournalNode's stored clusterID does not equal the clusterID in the NameNode's NamespaceInfo. The JN belongs to a different Hadoop cluster than the NN is claiming to be — a configuration or stale-format problem, never a transient fault.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java:275

        && state != StorageState.NON_EXISTENT
        && state != StorageState.NOT_FORMATTED;
    if (state == StorageState.NORMAL && startOpt != StartupOption.ROLLBACK) {
      readProperties(sd);
    } else if (needRecover) {
      sd.doRecover(state);
    }
  }

  void checkConsistentNamespace(NamespaceInfo nsInfo)
      throws IOException {
    if (nsInfo.getNamespaceID() != getNamespaceID()) {
      throw new IOException("Incompatible namespaceID for journal " +
          this.sd + ": NameNode has nsId " + nsInfo.getNamespaceID() +
          " but storage has nsId " + getNamespaceID());
    }
    
    if (!nsInfo.getClusterID().equals(getClusterID())) {
      throw new IOException("Incompatible clusterID for journal " +
          this.sd + ": NameNode has clusterId '" + nsInfo.getClusterID() +
          "' but storage has clusterId '" + getClusterID() + "'");
      
    }
  }

  public void close() throws IOException {
    LOG.info("Closing journal storage for {}", sd);
    unlockAll();
  }

  public boolean isFormatted() {
    return state == StorageState.NORMAL;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. If the NN format is the keeper: stop everything, wipe the JN journal dirs for this journal id, restart JNs, run 'hdfs namenode -initializeSharedEdits' to stamp the NN's clusterID/namespaceID on all JNs.
2. If the JN's cluster is the keeper (wrong NN pointing at it): fix the NN's clusterID or repoint dfs.namenode.shared.edits.dir to the correct JournalNodes.
3. Cross-check clusterID in the NN's and every JN's current/VERSION; they must all match.
4. Guard against recurrence: make reformatting a documented procedure that always includes JN dir cleanup.

// Preflight: clusterID must be identical on NN and every JN
String nnClusterId = NameNode.getNamespaceInfo().getClusterID();
for (URI jn : sharedEditsUris) {
  String jnClusterId = readJnVersionFile(jn).clusterID;
  if (!nnClusterId.equals(jnClusterId)) throw new IllegalStateException(
      "JN " + jn + " clusterId " + jnClusterId + " != NN " + nnClusterId);
}

static boolean isClusterIdMismatch(IOException ioe) {
  return ioe.getMessage() != null && ioe.getMessage().startsWith("Incompatible clusterID");
}

try {
  journalRpcServerCall();
} catch (IOException ioe) {
  if (isClusterIdMismatch(ioe)) {
    // identity conflict — retrying is pointless; reconcile formats/config
    haltAndAlertAdmin(ioe);
  } else {
    throw ioe;
  }
}

When it happens

Trigger: newEpoch from an NN whose clusterID differs from the JN's stored value: NN reformatted with a new auto-generated clusterID while JN dirs kept the old one; shared edits config pointing at a JN from another cluster; JN dir copied across environments (prod into test).

Common situations: Reformat of NN without re-initializing shared edits; environment cloning that copies JN data; editing clusterID by hand in one place only; multiple clusters sharing JN hosts with overlapping journal ids.

Related errors

• Incompatible namespaceID for journal {}: NameNode has nsId {
• Journal {} not formatted ; journal id: {}
• ACL '{}' not of expected form scheme:id
• Parsing {} :{}
• Unknown nameservice: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2fcba2d6b402b37e. Report an issue: GitHub.