What causes "Cannot find any valid remote NN to service request!" in apache/hadoop?

Check the active's actual state and health: 'hdfs haadmin -getServiceState ', the NN JMX State, and the active's log; wait out any in-progress failover

apache/hadoop · error · IOException

Cannot find any valid remote NN to service request!

Error message

Cannot find any valid remote NN to service request!

What it means

EditLogTailer on the standby iterates every remote NameNode and runs doWork() (rollEdits, counting most recent txns, tailing edits). Each IOException (connection refused, IPC timeout, StandbyException from a peer that is not active, SASL/Kerberos failure) nulls the cached proxy and advances to the next NN; when every remote NN has failed once, this IOException is thrown and the standby stays behind the active until a later tail cycle succeeds. The per-NN root causes are logged as 'Exception from remote name node ..., try next' just before the throw.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java:621

    protected abstract T doWork() throws IOException;

    public T call() throws IOException {
      // reset the loop count on success
      nnLoopCount = 0;
      while ((cachedActiveProxy = getActiveNodeProxy()) != null) {
        try {
          T ret = doWork();
          return ret;
        } catch (IOException e) {
          LOG.warn("Exception from remote name node " + currentNN
              + ", try next.", e);

          // Try next name node if exception happens.
          cachedActiveProxy = null;
          nnLoopCount++;
        }
      }
      throw new IOException("Cannot find any valid remote NN to service request!");
    }

    private NamenodeProtocol getActiveNodeProxy() throws IOException {
      if (cachedActiveProxy == null) {
        while (true) {
          // If the thread is interrupted, quit by returning null.
          if (Thread.currentThread().isInterrupted()) {
            LOG.warn("Interrupted while trying to getActiveNodeProxy.");
            return null;
          }

          // if we have reached the max loop count, quit by returning null
          if ((nnLoopCount / nnCount) >= maxRetries) {
            LOG.warn("Have reached the max loop count ({}).", nnLoopCount);
            return null;
          }

          currentNN = nnLookup.next();

View on GitHub (pinned to 2add963021)

Solutions

Check the active's actual state and health: 'hdfs haadmin -getServiceState <nn-id>', the NN JMX State, and the active's log; wait out any in-progress failover
From the standby, verify reachability of every remote NN RPC port (e.g. 'nc -z <host> 8020') and fix firewall/DNS/routing
Read the standby log above this exception: every 'Exception from remote name node ... try next' line carries the root cause (security, timeout, wrong state)
After fixing, confirm catch-up: the tailer's next cycle should log loaded edits, and 'Last applied transaction id' on the standby should converge with the active

Defensive patterns

Strategy: retry

Validate before calling

// Preflight: is the active NN reachable before expecting tailing to work
static boolean activeReachable(InetSocketAddress addr, int timeoutMs) {
  try (Socket s = new Socket()) {
    s.connect(addr, timeoutMs);
    return true;
  } catch (IOException e) {
    return false;
  }
}

Try / catch

// Around EditLogTailer-driven work (tail/roll) or code observing it:
try {
  tailOnce();
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Cannot find any valid remote NN")) {
    // every peer failed this round: log, alert if persistent, retry next cycle
    LOG.warn("edit tail round failed on all remote NNs; retrying next cycle", e);
  } else {
    throw e;
  }
}

Prevention

Monitor standby 'Last applied transaction id' lag and alert before it grows
Keep the per-NN 'Exception from remote name node ... try next' log lines indexed — the real cause is always there
Firewall/rules must allow standby->active on the RPC and HTTP ports permanently
Run 'hdfs haadmin -getServiceState' in health checks so both-standby states are caught early

When it happens

Trigger: All remote NameNodes fail during one EditLogTailer round: active NN down or restarting; firewall blocking the active's RPC port; wrong dfs.namenode.rpc-address configured on the standby side; both NNs in standby after a failed failover; Kerberos principal/keytab mismatch so the standby's RPC to the active is rejected; long active GC pause causing IPC timeouts.

Common situations: Failover in progress or split-brain being resolved; network partition between standby and active; security config drift between the two NNs; the tailer waking during an active restart window.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cb44f9e03ced9180. Report an issue: GitHub.