{"record":{"id":"cb44f9e03ced9180","repo":"apache/hadoop","slug":"cannot-find-any-valid-remote-nn-to-service-request","errorCode":null,"errorMessage":"Cannot find any valid remote NN to service request!","messagePattern":"Cannot find any valid remote NN to service request!","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java","lineNumber":621,"sourceCode":"    protected abstract T doWork() throws IOException;\n\n    public T call() throws IOException {\n      // reset the loop count on success\n      nnLoopCount = 0;\n      while ((cachedActiveProxy = getActiveNodeProxy()) != null) {\n        try {\n          T ret = doWork();\n          return ret;\n        } catch (IOException e) {\n          LOG.warn(\"Exception from remote name node \" + currentNN\n              + \", try next.\", e);\n\n          // Try next name node if exception happens.\n          cachedActiveProxy = null;\n          nnLoopCount++;\n        }\n      }\n      throw new IOException(\"Cannot find any valid remote NN to service request!\");\n    }\n\n    private NamenodeProtocol getActiveNodeProxy() throws IOException {\n      if (cachedActiveProxy == null) {\n        while (true) {\n          // If the thread is interrupted, quit by returning null.\n          if (Thread.currentThread().isInterrupted()) {\n            LOG.warn(\"Interrupted while trying to getActiveNodeProxy.\");\n            return null;\n          }\n\n          // if we have reached the max loop count, quit by returning null\n          if ((nnLoopCount / nnCount) >= maxRetries) {\n            LOG.warn(\"Have reached the max loop count ({}).\", nnLoopCount);\n            return null;\n          }\n\n          currentNN = nnLookup.next();","sourceCodeStart":603,"sourceCodeEnd":639,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java#L603-L639","documentation":"EditLogTailer on the standby iterates every remote NameNode and runs doWork() (rollEdits, counting most recent txns, tailing edits). Each IOException (connection refused, IPC timeout, StandbyException from a peer that is not active, SASL/Kerberos failure) nulls the cached proxy and advances to the next NN; when every remote NN has failed once, this IOException is thrown and the standby stays behind the active until a later tail cycle succeeds. The per-NN root causes are logged as 'Exception from remote name node ..., try next' just before the throw.","triggerScenarios":"All remote NameNodes fail during one EditLogTailer round: active NN down or restarting; firewall blocking the active's RPC port; wrong dfs.namenode.rpc-address configured on the standby side; both NNs in standby after a failed failover; Kerberos principal/keytab mismatch so the standby's RPC to the active is rejected; long active GC pause causing IPC timeouts.","commonSituations":"Failover in progress or split-brain being resolved; network partition between standby and active; security config drift between the two NNs; the tailer waking during an active restart window.","solutions":["Check the active's actual state and health: 'hdfs haadmin -getServiceState <nn-id>', the NN JMX State, and the active's log; wait out any in-progress failover","From the standby, verify reachability of every remote NN RPC port (e.g. 'nc -z <host> 8020') and fix firewall/DNS/routing","Read the standby log above this exception: every 'Exception from remote name node ... try next' line carries the root cause (security, timeout, wrong state)","After fixing, confirm catch-up: the tailer's next cycle should log loaded edits, and 'Last applied transaction id' on the standby should converge with the active"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Preflight: is the active NN reachable before expecting tailing to work\nstatic boolean activeReachable(InetSocketAddress addr, int timeoutMs) {\n  try (Socket s = new Socket()) {\n    s.connect(addr, timeoutMs);\n    return true;\n  } catch (IOException e) {\n    return false;\n  }\n}","typeGuard":null,"tryCatchPattern":"// Around EditLogTailer-driven work (tail/roll) or code observing it:\ntry {\n  tailOnce();\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"Cannot find any valid remote NN\")) {\n    // every peer failed this round: log, alert if persistent, retry next cycle\n    LOG.warn(\"edit tail round failed on all remote NNs; retrying next cycle\", e);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Monitor standby 'Last applied transaction id' lag and alert before it grows","Keep the per-NN 'Exception from remote name node ... try next' log lines indexed — the real cause is always there","Firewall/rules must allow standby->active on the RPC and HTTP ports permanently","Run 'hdfs haadmin -getServiceState' in health checks so both-standby states are caught early"],"tags":["hadoop","hdfs","high-availability","namenode","standby","network","edit-log"],"backgroundTag":"remote-peer-unreachable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}