What causes "Remote NameNodes not correctly configured!" in apache/hadoop?

Fix the HA topology in hdfs-site.xml: set dfs.ha.namenodes. to all NN ids and give every NN a dfs.namenode.rpc-address. . (and matching http address keys)

apache/hadoop · critical · IllegalArgumentException

Remote NameNodes not correctly configured!

Error message

Remote NameNodes not correctly configured!

What it means

Thrown by the standby NameNode's EditLogTailer constructor. When dfs.ha.log-roll.period >= 0 (default 120s), the tailer must find the other NameNodes in the nameservice to roll and tail their edit logs; RemoteNameNodeInfo.getRemoteNameNodes(conf) re-reads the HA topology (dfs.nameservices / dfs.internal.nameservices, dfs.ha.namenodes.<ns>, dfs.namenode.rpc-address.<ns>.<nn>) and throws IOException when it cannot resolve the peer NNs. The tailer wraps that IOException in IllegalArgumentException, which aborts NameNode startup or the transition to standby.

Solutions

Fix the HA topology in hdfs-site.xml: set dfs.ha.namenodes.<ns-id> to all NN ids and give every NN a dfs.namenode.rpc-address.<ns-id>.<nn-id> (and matching http address keys)
Verify the config as the NameNode resolves it: 'hdfs getconf -dfs.nameservices', 'hdfs getconf -dfs.ha.namenodes' and confirm each address resolves (host -t A <host>)
Fix hostname/DNS or /etc/hosts for every listed NN address if resolution is the failure
Only if you deliberately want no edit-log tailing (not recommended in HA): set dfs.ha.log-roll.period to a negative value so remote-NN discovery is skipped

Example fix

<!-- before: nn2 has no address -->
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>nn1-host:8020</value></property>
<!-- after -->
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>nn1-host:8020</value></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn2</name><value>nn2-host:8020</value></property>

Defensive patterns

Strategy: validation

Validate before calling

// Run before NN start / transitionToStandby on this host's effective config
static void validateRemoteNameNodes(Configuration conf) throws IOException {
  String nsId = DFSUtil.getNamenodeNameServiceId(conf);
  if (nsId == null) return; // single-NN, no federation: nothing to check
  String idsKey = "dfs.ha.namenodes." + nsId;
  String[] nnIds = conf.getTrimmedStrings(idsKey);
  if (nnIds == null || nnIds.length == 0)
    throw new IOException(idsKey + " is not configured");
  for (String nnId : nnIds) {
    String addrKey = "dfs.namenode.rpc.address." + nsId + "." + nnId;
    String addr = conf.get(addrKey);
    if (addr == null || addr.isEmpty())
      throw new IOException(addrKey + " is not configured");
    InetAddress.getByName(addr.split(":")[0]); // fails on unresolvable host
  }
}

Prevention

Template HA configs with the <ns-id>.<nn-id> suffixed address keys so a missing suffix cannot happen
Add a pre-deploy config lint (hdfs getconf + a key-presence check) to your provisioning pipeline
Resolve all NN hostnames on every node before starting or failing over
Keep both NNs' hdfs-site.xml byte-identical except where deployment tooling intentionally differs

When it happens

Trigger: Starting a NameNode (or transitioning it to standby) while log rolling is enabled and: (a) this NN's nameservice id has no dfs.ha.namenodes.<ns-id> entry, (b) an NN id listed there has no dfs.namenode.rpc-address.<ns-id>.<nn-id> so NameNode.getServiceAddress cannot build a socket address, or (c) a host in one of those addresses is unresolvable, making HAUtil.getConfForOtherNodes fail.

Common situations: Hand-edited hdfs-site.xml with a typo in dfs.ha.namenodes.* or a missing per-NN address suffix; cloning one node's config to another without the <ns-id>.<nn-id> suffixed keys; federation misconfiguration of dfs.internal.nameservices; DNS entries for NN hosts missing.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/23d397e827bc4638. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java:206

  public EditLogTailer(FSNamesystem namesystem, Configuration conf) {
    this.tailerThread = new EditLogTailerThread();
    this.conf = conf;
    this.namesystem = namesystem;
    this.timer = new Timer();
    this.editLog = namesystem.getEditLog();
    this.lastLoadTimeMs = timer.monotonicNow();
    this.lastRollTimeMs = timer.monotonicNow();

    logRollPeriodMs = conf.getTimeDuration(
        DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_KEY,
        DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_DEFAULT,
        TimeUnit.SECONDS, TimeUnit.MILLISECONDS);
    List<RemoteNameNodeInfo> nns = Collections.emptyList();
    if (logRollPeriodMs >= 0) {
      try {
        nns = RemoteNameNodeInfo.getRemoteNameNodes(conf);
      } catch (IOException e) {
        throw new IllegalArgumentException("Remote NameNodes not correctly configured!", e);
      }

      for (RemoteNameNodeInfo info : nns) {
        // overwrite the socket address, if we need to
        InetSocketAddress ipc = NameNode.getServiceAddress(info.getConfiguration(), true);
        // sanity check the ipc address
        Preconditions.checkArgument(ipc.getPort() > 0,
            "Active NameNode must have an IPC port configured. " + "Got address '%s'", ipc);
        info.setIpcAddress(ipc);
      }

      LOG.info("Will roll logs on active node every " +
          (logRollPeriodMs / 1000) + " seconds.");
    } else {
      LOG.info("Not going to trigger log rolls on active node because " +
          DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_KEY + " is negative.");
    }

View on GitHub (pinned to 2add963021)