{"record":{"id":"4651d71357eafdce","repo":"apache/hadoop","slug":"idhelptext-mismatch-previously-connected","errorCode":null,"errorMessage":"\" + idHelpText + \" mismatch: previously connected to \" + idHelpText + \" \" + ourID + \" but now connected to \" + idHelpText + \" \" + theirID","messagePattern":"\" \\+ idHelpText \\+ \" mismatch: previously connected to \" \\+ idHelpText \\+ \" \" \\+ ourID \\+ \" but now connected to \" \\+ idHelpText \\+ \" \" \\+ theirID","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java","lineNumber":455,"sourceCode":"        boolean updateCurrentKey = bpServiceActor.state == null\n            || bpServiceActor.state == HAServiceState.ACTIVE;\n        dn.blockPoolTokenSecretManager.addKeys(getBlockPoolId(),\n            reg.getExportedKeys(), updateCurrentKey);\n      }\n    } finally {\n      writeUnlock();\n    }\n  }\n\n  /**\n   * Verify equality of two namespace-related fields, throwing\n   * an exception if they are unequal.\n   */\n  private static void checkNSEquality(\n      Object ourID, Object theirID,\n      String idHelpText) throws IOException {\n    if (!ourID.equals(theirID)) {\n      throw new IOException(idHelpText + \" mismatch: \" +\n          \"previously connected to \" + idHelpText + \" \" + ourID + \n          \" but now connected to \" + idHelpText + \" \" + theirID);\n    }\n  }\n\n  DatanodeRegistration createRegistration() {\n    writeLock();\n    try {\n      Preconditions.checkState(bpNSInfo != null,\n          \"getRegistration() can only be called after initial handshake\");\n      return dn.createBPRegistration(bpNSInfo);\n    } finally {\n      writeUnlock();\n    }\n  }\n\n  /**\n   * Called when an actor shuts down. If this is the last actor","sourceCodeStart":437,"sourceCodeEnd":473,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java#L437-L473","documentation":"A BPOfferService serves exactly one namespace and remembers the identity (blockPoolID, namespaceID, clusterID) reported by the first NameNode it talked to. checkNSEquality throws when a later response from an NN of the same namespace (the HA active/standby pair, or a re-registering NN) reports a different value — the DN refuses to serve two identities under one block pool, which would corrupt block metadata.","triggerScenarios":"registrationSucceeded/namespace-info handling comparing bpNSInfo against a freshly received nsInfo: blockPoolID (line 251), namespaceID (253), or clusterID (255) differing between the two NNs the DN believes belong to the same namespace.","commonSituations":"NameNode was reformatted (new clusterID/namespaceID) while DN storage kept the old ID; HA addresses misconfigured so both HA entries point at NNs of different namespaces; federation nameservices sharing/overlapping address config; DN data dirs reused from another cluster.","solutions":["Identify which value flipped (clusterID vs namespaceID vs blockPoolID are printed) and compare NN and DN VERSION files to see which side is stale","If the NN was deliberately reformatted, also clear/reformat the DataNode's storage for that block pool (or the whole data dirs) — never reuse DN dirs across reformats","Fix HA/federation config: every nameservice's NN pair (dfs.ha.namenodes.<id> with their rpc/service-rpc addresses) must be the active/standby of one namespace","If DNs are stale but data must be kept, use the clusterID-matching reformat path rather than mixing IDs"],"exampleFix":"# before: DN dirs keep old clusterID after NN reformat\n# NN VERSION: clusterID=CID-new   DN VERSION: clusterID=CID-old\n\n# after: recluster the DN for that block pool\nstop datanode\ncat /data/dn/current/BP-*/VERSION   # inspect\n# either wipe /data/dn/current (data loss) or reformat NN back to CID-old","handlingStrategy":"validation","validationCode":"# Operator preflight: NN and DN must agree before the DN starts\nnn_cid=$(grep -oP 'clusterID=\\K.*' $NN_VERSION_FILE)\nfor v in /data/dn/current/BP-*/VERSION; do\n  dn_cid=$(grep -oP 'clusterID=\\K.*' $v)\n  [ \"$nn_cid\" = \"$dn_cid\" ] || echo \"MISMATCH: $v has $dn_cid, NN has $nn_cid\"\ndone","typeGuard":null,"tryCatchPattern":null,"preventionTips":["After any NN reformat, always clean or reformat DN storage — never mix IDs","Automate VERSION-file clusterID checks in cluster bring-up scripts","Review HA/federation address config so each nameservice has exactly one namespace behind it"],"tags":["hdfs","datanode","namenode","clusterid","namespaceid","ha","reformat"],"backgroundTag":"cluster-id-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}