apache/hadoop · critical · RemoteException

No valid proxies left. All NameNode proxies have failed over

Error message

No valid proxies left. All NameNode proxies have failed over.

What it means

RequestHedgingProxyProvider strips the proxy that previously failed over (toIgnore) from its candidate map before each invocation. When that removal leaves zero targets, it throws RemoteException wrapping IOException 'No valid proxies left' - every configured NameNode has already been marked failed-over, so the hedging provider has no endpoint left to call.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java:104

            throws Throwable {
      // Need double check locking to guarantee thread-safe since
      // currentUsedProxy is lazily initialized.
      if (currentUsedProxy == null) {
        synchronized (this) {
          if (currentUsedProxy == null) {
            Map<Future<Object>, ProxyInfo<T>> proxyMap = new HashMap<>();
            int numAttempts = 0;

            ExecutorService executor = null;
            CompletionService<Object> completionService;
            try {
              // Optimization : if only 2 proxies are configured and one had
              // failed
              // over, then we dont need to create a threadpool etc.
              targetProxies.remove(toIgnore);
              if (targetProxies.size() == 0) {
                LOG.trace("No valid proxies left");
                throw new RemoteException(IOException.class.getName(),
                    "No valid proxies left. "
                        + "All NameNode proxies have failed over.");
              }
              if (targetProxies.size() == 1) {
                ProxyInfo<T> proxyInfo =
                    targetProxies.values().iterator().next();
                try {
                  currentUsedProxy = proxyInfo;
                  Object retVal = method.invoke(proxyInfo.proxy, args);
                  LOG.debug("Invocation successful on [{}]",
                      currentUsedProxy.proxyInfo);
                  return retVal;
                } catch (InvocationTargetException ex) {
                  Exception unwrappedException =
                      unwrapInvocationTargetException(ex);
                  logProxyException(unwrappedException,
                      currentUsedProxy.proxyInfo);
                  LOG.trace("Unsuccessful invocation on [{}]",

View on GitHub (pinned to 2add963021)

Solutions

1. Verify NameNode health: 'hdfs haadmin -getAllServiceState' - one must be active; check ZKFC and JournalNode processes
2. Fix client-to-NameNode connectivity (RPC ports, firewalls, DNS)
3. Restart failed NameNodes/ZKFCs so failover can complete
4. Retry the client operation after recovery - a fresh invocation repopulates the proxy map

catch (RemoteException e) {
  if (IOException.class.getName().equals(e.getClassName())
      && e.getMessage() != null
      && e.getMessage().contains("No valid proxies left")) {
    // all NameNodes down: back off, check cluster health, then retry the op
    backoffAndCheckNameNodes();
    return retryWithNewFileSystem();
  }
  throw e;
}

When it happens

Trigger: All NameNodes in the nameservice are down or unreachable (crash, both stuck in standby, ZKFC/journal problems, network partition) while successive hedged invocations keep failing proxies over until targetProxies is empty.

Common situations: Cluster-wide NameNode outage; switch/rack failure isolating the client; both NNs in standby due to shared-journal (QJM) issues; aggressive hedging retaining stale failure state.

Related errors

• Invalid configuration: a shared edits dir must not be specif
• Remote NameNodes not correctly configured!
• Cannot find any valid remote NN to service request!
• Transition from state {} to {} is not allowed.
• Cannot finalize with no NameNode active

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/051d265648903080. Report an issue: GitHub.