{"record":{"id":"051d265648903080","repo":"apache/hadoop","slug":"no-valid-proxies-left-all-namenode-proxies-have-f","errorCode":null,"errorMessage":"No valid proxies left. All NameNode proxies have failed over.","messagePattern":"No valid proxies left\\. All NameNode proxies have failed over\\.","errorType":"exception","errorClass":"RemoteException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java","lineNumber":104,"sourceCode":"            throws Throwable {\n      // Need double check locking to guarantee thread-safe since\n      // currentUsedProxy is lazily initialized.\n      if (currentUsedProxy == null) {\n        synchronized (this) {\n          if (currentUsedProxy == null) {\n            Map<Future<Object>, ProxyInfo<T>> proxyMap = new HashMap<>();\n            int numAttempts = 0;\n\n            ExecutorService executor = null;\n            CompletionService<Object> completionService;\n            try {\n              // Optimization : if only 2 proxies are configured and one had\n              // failed\n              // over, then we dont need to create a threadpool etc.\n              targetProxies.remove(toIgnore);\n              if (targetProxies.size() == 0) {\n                LOG.trace(\"No valid proxies left\");\n                throw new RemoteException(IOException.class.getName(),\n                    \"No valid proxies left. \"\n                        + \"All NameNode proxies have failed over.\");\n              }\n              if (targetProxies.size() == 1) {\n                ProxyInfo<T> proxyInfo =\n                    targetProxies.values().iterator().next();\n                try {\n                  currentUsedProxy = proxyInfo;\n                  Object retVal = method.invoke(proxyInfo.proxy, args);\n                  LOG.debug(\"Invocation successful on [{}]\",\n                      currentUsedProxy.proxyInfo);\n                  return retVal;\n                } catch (InvocationTargetException ex) {\n                  Exception unwrappedException =\n                      unwrapInvocationTargetException(ex);\n                  logProxyException(unwrappedException,\n                      currentUsedProxy.proxyInfo);\n                  LOG.trace(\"Unsuccessful invocation on [{}]\",","sourceCodeStart":86,"sourceCodeEnd":122,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java#L86-L122","documentation":"RequestHedgingProxyProvider strips the proxy that previously failed over (toIgnore) from its candidate map before each invocation. When that removal leaves zero targets, it throws RemoteException wrapping IOException 'No valid proxies left' - every configured NameNode has already been marked failed-over, so the hedging provider has no endpoint left to call.","triggerScenarios":"All NameNodes in the nameservice are down or unreachable (crash, both stuck in standby, ZKFC/journal problems, network partition) while successive hedged invocations keep failing proxies over until targetProxies is empty.","commonSituations":"Cluster-wide NameNode outage; switch/rack failure isolating the client; both NNs in standby due to shared-journal (QJM) issues; aggressive hedging retaining stale failure state.","solutions":["Verify NameNode health: 'hdfs haadmin -getAllServiceState' - one must be active; check ZKFC and JournalNode processes","Fix client-to-NameNode connectivity (RPC ports, firewalls, DNS)","Restart failed NameNodes/ZKFCs so failover can complete","Retry the client operation after recovery - a fresh invocation repopulates the proxy map"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (RemoteException e) {\n  if (IOException.class.getName().equals(e.getClassName())\n      && e.getMessage() != null\n      && e.getMessage().contains(\"No valid proxies left\")) {\n    // all NameNodes down: back off, check cluster health, then retry the op\n    backoffAndCheckNameNodes();\n    return retryWithNewFileSystem();\n  }\n  throw e;\n}","preventionTips":["Monitor both NameNodes and ZKFC; alert before both fail simultaneously","Wrap hedged-client calls with application-level retry and backoff","Keep 'hdfs haadmin -getAllServiceState' in runbooks for fast triage"],"tags":["hdfs","high-availability","hedging","name-node-unavailable"],"backgroundTag":"all-instances-unavailable","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}