apache/hadoop · critical · IOException

Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCi

Error message

Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCid}; bpid={bpid}

What it means

During block-pool handshake the DataNode compares its persisted clusterId (VERSION file under its storage dirs) with the clusterId the NameNode namespace announces (nsCid); setClusterId throws IOException when they differ, so the DN refuses to serve blocks for a foreign namespace. The bpid in the message identifies which block pool triggered the mismatch.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1444

    // Set configuration and dataDirs to reflect volume changes.
    for (Iterator<StorageLocation> it = dataDirs.iterator(); it.hasNext(); ) {
      StorageLocation loc = it.next();
      if (storageLocations.contains(loc)) {
        it.remove();
      }
    }
    getConf().set(DFS_DATANODE_DATA_DIR_KEY, Joiner.on(",").join(dataDirs));

    if (ioe != null) {
      throw ioe;
    }
  }

  private synchronized void setClusterId(final String nsCid, final String bpid
      ) throws IOException {
    if(clusterId != null && !clusterId.equals(nsCid)) {
      throw new IOException ("Cluster IDs not matched: dn cid=" + clusterId 
          + " but ns cid="+ nsCid + "; bpid=" + bpid);
    }
    // else
    clusterId = nsCid;
  }

  /**
   * Returns the hostname for this datanode. If the hostname is not
   * explicitly configured in the given config, then it is determined
   * via the DNS class.
   *
   * @param config configuration
   * @return the hostname (NB: may not be a FQDN)
   * @throws UnknownHostException if the dfs.datanode.dns.interface
   *    option is used and the hostname can not be determined
   */
  private static String getHostName(Configuration config)
      throws UnknownHostException {

View on GitHub (pinned to 2add963021)

Solutions

1. If the NN reformat was intentional, reset DN storage: stop the DN, remove or move aside the contents of its dfs.datanode.data.dir directories, then start it — it re-registers with the new clusterID
2. To preserve DN data, instead re-format the NN pinned to the old ID: hdfs namenode -format -clusterId <OLD_CLUSTER_ID>
3. If it is a config slip, fix dfs.nameservices / dfs.ha.namenodes / dfs.namenode.rpc-address so the DN talks to its own namespace

Example fix

# before: NN reformatted, DN still holds old VERSION -> Cluster IDs not matched
# after
$ hdfs --daemon stop datanode
$ sudo -u hdfs rm -rf /data/dn*/current/*   # backup first if data matters
$ hdfs --daemon start datanode

# Before starting/joining the DN, compare IDs:
# NN:  curl -s http://nn:9870/jmx | jq '.beans[] | select(.name=="Hadoop:service=NameNode,name=NameNodeStatus") | .ClusterId'
# DN:  grep clusterID /data/dn/current/VERSION
# Abort the start if the strings differ.

try {
  dn.runDatanodeDaemon(); // setClusterId fires during BP handshake
} catch (IOException e) {
  if (e.getMessage().contains("Cluster IDs not matched")) {
    // decide: wipe DN storage (fresh NN) or reformat NN with -clusterId <old>
  }
}

When it happens

Trigger: NameNode was reformatted (fresh clusterID) while DataNode dirs keep the old VERSION; DN pointed at a different federation namespace via wrong dfs.nameservices / dfs.namenode.rpc-address; an HA pair rebuilt from scratch while DNs retain data from the old cluster.

Common situations: Re-running 'hdfs namenode -format' on a test cluster without cleaning DN dirs; swapping a test NN address for a prod one in DN configs; cloning VMs with stale storage directories.

Related errors

• Although a UNIX domain socket path is configured as {domainS
• Cannot start datanode because the configured max locked memo
• Invalid value configured for dfs.datanode.failed.volumes.tol
• Security is enabled but block access tokens (via dfs.block.a
• Storage not yet initialized for {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3b351cc5907b4d13. Report an issue: GitHub.