apache/hadoop · error · HadoopIllegalArgumentException

Could not get the namenode ID of this node. You may run zkfc

Error message

Could not get the namenode ID of this node. You may run zkfc on the node other than namenode.

What it means

After confirming HA is enabled, DFSZKFailoverController.create asks HAUtil.getNameNodeId(localNNConf, nsId) which NameNode this host is. Resolution checks dfs.ha.namenode.id in the config and falls back to matching dfs.namenode.rpc-address.<ns>.<nn> hostnames against the local address. A null result means the node was not identified as any configured NameNode, so the ZKFC cannot know which NameNode to guard and rejects startup. The message explicitly suggests zkfc is running on a non-NameNode machine.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSZKFailoverController.java:160

    if (addr == null || addr.isEmpty()) {
      addr = conf.getTrimmed(DFSConfigKeys.DFS_NAMENODE_RPC_BIND_HOST_KEY);
    }
    return addr;
  }

  public static DFSZKFailoverController create(Configuration conf) {
    Configuration localNNConf = DFSHAAdmin.addSecurityConfiguration(conf);
    String nsId = DFSUtil.getNamenodeNameServiceId(conf);

    if (!HAUtil.isHAEnabled(localNNConf, nsId)) {
      throw new HadoopIllegalArgumentException(
          "HA is not enabled for this namenode.");
    }
    String nnId = HAUtil.getNameNodeId(localNNConf, nsId);
    if (nnId == null) {
      String msg = "Could not get the namenode ID of this node. " +
          "You may run zkfc on the node other than namenode.";
      throw new HadoopIllegalArgumentException(msg);
    }
    NameNode.initializeGenericKeys(localNNConf, nsId, nnId);
    DFSUtil.setGenericConf(localNNConf, nsId, nnId, ZKFC_CONF_KEYS);
    
    NNHAServiceTarget localTarget = new NNHAServiceTarget(
        localNNConf, nsId, nnId);
    return new DFSZKFailoverController(localNNConf, localTarget);
  }

  private DFSZKFailoverController(Configuration conf,
      NNHAServiceTarget localTarget) {
    super(conf, localTarget);
    this.localNNTarget = localTarget;
    // Setup ACLs
    adminAcl = new AccessControlList(
        conf.get(DFSConfigKeys.DFS_ADMIN, " "));
    LOG.info("Failover controller configured for NameNode " +
        localTarget);

View on GitHub (pinned to 2add963021)

Solutions

On the NameNode host, set dfs.ha.namenode.id (e.g. nn1) in hdfs-site.xml and restart zkfc.
If dfs.ha.namenode.id is intentionally absent, make the local hostname resolve exactly to the host part of one dfs.namenode.rpc-address.<ns>.<nn> entry (check 'hostname -f' vs the config value).
Confirm zkfc is actually supposed to run here: it must run on every NameNode and only on NameNodes.
Re-run and confirm the next lines (service address resolution, fencer check) pass.

Example fix

<!-- before: NN-specific id missing, hostname matching fails -->
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property>

<!-- after: this host claims its NN id -->
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property>
<property><name>dfs.ha.namenode.id</name><value>nn1</value></property>

Defensive patterns

Strategy: validation

Validate before calling

Configuration conf = DFSHAAdmin.addSecurityConfiguration(baseConf);
String nsId = DFSUtil.getNamenodeNameServiceId(conf);
if (HAUtil.isHAEnabled(conf, nsId)
    && HAUtil.getNameNodeId(conf, nsId) == null) {
  throw new IllegalStateException("This host is not a configured NameNode of "
      + nsId + "; set dfs.ha.namenode.id or fix hostname/rpc-address matching.");
}

Try / catch

try {
  DFSZKFailoverController.create(conf);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("namenode ID")) {
    // host identity problem: check dfs.ha.namenode.id and hostname resolution, not code
    LOG.error("zkfc cannot identify local NameNode: {}", e.getMessage());
  }
  throw e;
}

Prevention

Set dfs.ha.namenode.id explicitly per NameNode host instead of relying on hostname matching.
Verify 'hostname -f' output equals the host part of a dfs.namenode.rpc-address.<ns>.<nn> entry.
Only provision zkfc on hosts listed as HA NameNodes.

When it happens

Trigger: Starting zkfc on a host that is not one of the HA NameNodes; dfs.ha.namenode.id missing from hdfs-site.xml and the host's hostname/reverse-DNS not matching any dfs.namenode.rpc-address.<ns>.<nn> entry; hostname resolving to a short name (host1) while the config uses an FQDN (host1.example.com).

Common situations: ZKFC deployed by a generic cluster-wide config rollout where each NN was supposed to get its own dfs.ha.namenode.id; /etc/hosts or DNS giving a different name than what is in the address keys; running zkfc on a standby gateway node by mistake.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/98b163844e762bcc. Report an issue: GitHub.