apache/hadoop · critical · IOException

" + idHelpText + " mismatch: previously connected to " + idH

Error message

" + idHelpText + " mismatch: previously connected to " + idHelpText + " " + ourID + " but now connected to " + idHelpText + " " + theirID

What it means

A BPOfferService serves exactly one namespace and remembers the identity (blockPoolID, namespaceID, clusterID) reported by the first NameNode it talked to. checkNSEquality throws when a later response from an NN of the same namespace (the HA active/standby pair, or a re-registering NN) reports a different value — the DN refuses to serve two identities under one block pool, which would corrupt block metadata.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java:455

        boolean updateCurrentKey = bpServiceActor.state == null
            || bpServiceActor.state == HAServiceState.ACTIVE;
        dn.blockPoolTokenSecretManager.addKeys(getBlockPoolId(),
            reg.getExportedKeys(), updateCurrentKey);
      }
    } finally {
      writeUnlock();
    }
  }

  /**
   * Verify equality of two namespace-related fields, throwing
   * an exception if they are unequal.
   */
  private static void checkNSEquality(
      Object ourID, Object theirID,
      String idHelpText) throws IOException {
    if (!ourID.equals(theirID)) {
      throw new IOException(idHelpText + " mismatch: " +
          "previously connected to " + idHelpText + " " + ourID + 
          " but now connected to " + idHelpText + " " + theirID);
    }
  }

  DatanodeRegistration createRegistration() {
    writeLock();
    try {
      Preconditions.checkState(bpNSInfo != null,
          "getRegistration() can only be called after initial handshake");
      return dn.createBPRegistration(bpNSInfo);
    } finally {
      writeUnlock();
    }
  }

  /**
   * Called when an actor shuts down. If this is the last actor

View on GitHub (pinned to 2add963021)

Solutions

1. Identify which value flipped (clusterID vs namespaceID vs blockPoolID are printed) and compare NN and DN VERSION files to see which side is stale
2. If the NN was deliberately reformatted, also clear/reformat the DataNode's storage for that block pool (or the whole data dirs) — never reuse DN dirs across reformats
3. Fix HA/federation config: every nameservice's NN pair (dfs.ha.namenodes.<id> with their rpc/service-rpc addresses) must be the active/standby of one namespace
4. If DNs are stale but data must be kept, use the clusterID-matching reformat path rather than mixing IDs

Example fix

# before: DN dirs keep old clusterID after NN reformat
# NN VERSION: clusterID=CID-new   DN VERSION: clusterID=CID-old

# after: recluster the DN for that block pool
stop datanode
cat /data/dn/current/BP-*/VERSION   # inspect
# either wipe /data/dn/current (data loss) or reformat NN back to CID-old

# Operator preflight: NN and DN must agree before the DN starts
nn_cid=$(grep -oP 'clusterID=\K.*' $NN_VERSION_FILE)
for v in /data/dn/current/BP-*/VERSION; do
  dn_cid=$(grep -oP 'clusterID=\K.*' $v)
  [ "$nn_cid" = "$dn_cid" ] || echo "MISMATCH: $v has $dn_cid, NN has $nn_cid"
done

When it happens

Trigger: registrationSucceeded/namespace-info handling comparing bpNSInfo against a freshly received nsInfo: blockPoolID (line 251), namespaceID (253), or clusterID (255) differing between the two NNs the DN believes belong to the same namespace.

Common situations: NameNode was reformatted (new clusterID/namespaceID) while DN storage kept the old ID; HA addresses misconfigured so both HA entries point at NNs of different namespaces; federation nameservices sharing/overlapping address config; DN data dirs reused from another cluster.

Related errors

• Block pool {bpid} has not recognized an active NN
• Incompatible clusterIDs in {}: namenode clusterID = {}; data
• Configuration has multiple addresses that match local node's
• Configuration dfs.namenode.rpc-address must be suffixed with
• Unexpected HAServiceStateProto:

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4651d71357eafdce. Report an issue: GitHub.