apache/hadoop · critical · IllegalArgumentException

Remote NameNodes not correctly configured!

Error message

Remote NameNodes not correctly configured!

What it means

Thrown by the standby NameNode's EditLogTailer constructor. When dfs.ha.log-roll.period >= 0 (default 120s), the tailer must find the other NameNodes in the nameservice to roll and tail their edit logs; RemoteNameNodeInfo.getRemoteNameNodes(conf) re-reads the HA topology (dfs.nameservices / dfs.internal.nameservices, dfs.ha.namenodes.<ns>, dfs.namenode.rpc-address.<ns>.<nn>) and throws IOException when it cannot resolve the peer NNs. The tailer wraps that IOException in IllegalArgumentException, which aborts NameNode startup or the transition to standby.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java:206

  public EditLogTailer(FSNamesystem namesystem, Configuration conf) {
    this.tailerThread = new EditLogTailerThread();
    this.conf = conf;
    this.namesystem = namesystem;
    this.timer = new Timer();
    this.editLog = namesystem.getEditLog();
    this.lastLoadTimeMs = timer.monotonicNow();
    this.lastRollTimeMs = timer.monotonicNow();

    logRollPeriodMs = conf.getTimeDuration(
        DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_KEY,
        DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_DEFAULT,
        TimeUnit.SECONDS, TimeUnit.MILLISECONDS);
    List<RemoteNameNodeInfo> nns = Collections.emptyList();
    if (logRollPeriodMs >= 0) {
      try {
        nns = RemoteNameNodeInfo.getRemoteNameNodes(conf);
      } catch (IOException e) {
        throw new IllegalArgumentException("Remote NameNodes not correctly configured!", e);
      }

      for (RemoteNameNodeInfo info : nns) {
        // overwrite the socket address, if we need to
        InetSocketAddress ipc = NameNode.getServiceAddress(info.getConfiguration(), true);
        // sanity check the ipc address
        Preconditions.checkArgument(ipc.getPort() > 0,
            "Active NameNode must have an IPC port configured. " + "Got address '%s'", ipc);
        info.setIpcAddress(ipc);
      }

      LOG.info("Will roll logs on active node every " +
          (logRollPeriodMs / 1000) + " seconds.");
    } else {
      LOG.info("Not going to trigger log rolls on active node because " +
          DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_KEY + " is negative.");
    }
    

View on GitHub (pinned to 2add963021)

Solutions

1. Fix the HA topology in hdfs-site.xml: set dfs.ha.namenodes.<ns-id> to all NN ids and give every NN a dfs.namenode.rpc-address.<ns-id>.<nn-id> (and matching http address keys)
2. Verify the config as the NameNode resolves it: 'hdfs getconf -dfs.nameservices', 'hdfs getconf -dfs.ha.namenodes' and confirm each address resolves (host -t A <host>)
3. Fix hostname/DNS or /etc/hosts for every listed NN address if resolution is the failure
4. Only if you deliberately want no edit-log tailing (not recommended in HA): set dfs.ha.log-roll.period to a negative value so remote-NN discovery is skipped

Example fix

<!-- before: nn2 has no address -->
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>nn1-host:8020</value></property>
<!-- after -->
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>nn1-host:8020</value></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn2</name><value>nn2-host:8020</value></property>

// Run before NN start / transitionToStandby on this host's effective config
static void validateRemoteNameNodes(Configuration conf) throws IOException {
  String nsId = DFSUtil.getNamenodeNameServiceId(conf);
  if (nsId == null) return; // single-NN, no federation: nothing to check
  String idsKey = "dfs.ha.namenodes." + nsId;
  String[] nnIds = conf.getTrimmedStrings(idsKey);
  if (nnIds == null || nnIds.length == 0)
    throw new IOException(idsKey + " is not configured");
  for (String nnId : nnIds) {
    String addrKey = "dfs.namenode.rpc.address." + nsId + "." + nnId;
    String addr = conf.get(addrKey);
    if (addr == null || addr.isEmpty())
      throw new IOException(addrKey + " is not configured");
    InetAddress.getByName(addr.split(":")[0]); // fails on unresolvable host
  }
}

When it happens

Trigger: Starting a NameNode (or transitioning it to standby) while log rolling is enabled and: (a) this NN's nameservice id has no dfs.ha.namenodes.<ns-id> entry, (b) an NN id listed there has no dfs.namenode.rpc-address.<ns-id>.<nn-id> so NameNode.getServiceAddress cannot build a socket address, or (c) a host in one of those addresses is unresolvable, making HAUtil.getConfForOtherNodes fail.

Common situations: Hand-edited hdfs-site.xml with a typo in dfs.ha.namenodes.* or a missing per-NN address suffix; cloning one node's config to another without the <ns-id>.<nn-id> suffixed keys; federation misconfiguration of dfs.internal.nameservices; DNS entries for NN hosts missing.

Related errors

• Required edits directory {} not found: dfs.namenode.edits.di
• Invalid configuration: a shared edits dir must not be specif
• Cannot find any valid remote NN to service request!
• Unknown nameservice: {}
• Configuration has multiple addresses that match local node's

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/23d397e827bc4638. Report an issue: GitHub.