{"record":{"id":"c3b0345ae1f69e20","repo":"apache/hadoop","slug":"namespace-nsid-async-handler-is-busy","errorCode":null,"errorMessage":"Namespace '{nsid}' async handler is busy.","messagePattern":"Namespace '(.+?)' async handler is busy\\.","errorType":"exception","errorClass":"StandbyException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/async/RouterAsyncRpcClient.java","lineNumber":199,"sourceCode":"    asyncApplyUseExecutor((AsyncApplyFunction<Object, Object>) o -> {\n      if (LOG.isDebugEnabled()) {\n        LOG.debug(\"Async invoke method : {}, {}, {}, {}\", method.getName(), useObserver, namenodes,\n            params);\n      }\n      threadLocalContext.transfer();\n      RouterRpcFairnessPolicyController controller = getRouterRpcFairnessPolicyController();\n      acquirePermit(nsid, ugi, method.getName(), controller);\n      invokeMethodAsync(ugi, (List<FederationNamenodeContext>) namenodes,\n          useObserver, protocol, method, params);\n      asyncFinally(object -> {\n        releasePermit(nsid, ugi, method, controller);\n        return object;\n      });\n    }, router.getRpcServer().getAsyncExecutorForNamespace(nsid));\n\n    // Catch the RejectedExecutionException and convert it to StandbyException\n    asyncCatch((ret, e) -> {\n      throw new StandbyException(\"Namespace '\" + nsid + \"' async handler is busy.\");\n    }, RejectedExecutionException.class);\n    return null;\n  }\n\n  /**\n   * Asynchronously invokes a method on the specified NameNodes for a given user and operation.\n   * This method is responsible for the actual execution of the remote method call on the\n   * NameNodes in a non-blocking manner, allowing for concurrent processing.\n   *\n   * <p>In case of exceptions, the method includes logic to handle retries, failover to standby\n   * NameNodes, and proper exception handling to ensure that the calling code can respond\n   * appropriately to different error conditions.\n   *\n   * @param ugi The user information under which the method is to be invoked.\n   * @param namenodes The list of NameNode contexts on which the method will be invoked.\n   * @param useObserver Whether to use an observer node for the invocation if available.\n   * @param protocol The protocol class defining the method to be invoked.\n   * @param method The method to be invoked on the NameNodes.","sourceCodeStart":181,"sourceCodeEnd":217,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/async/RouterAsyncRpcClient.java#L181-L217","documentation":"A StandbyException raised when the per-nameservice asynchronous handler thread pool rejects a task. RouterRpcServer creates one bounded ThreadPoolExecutor per nameservice (handlers = dfs.federation.router.async.rpc.handler.count, default 10, or per-ns dfs.federation.router.async.rpc.ns.handler.count; queue = LinkedBlockingQueue of dfs.federation.router.async.rpc.queue.size, default 1000). When the queue is full, asyncApplyUseExecutor completes exceptionally with RejectedExecutionException, and asyncCatch converts it into StandbyException so clients treat the Router like a standby Namenode and retry. This is intentional backpressure to prevent unbounded memory growth.","triggerScenarios":"A burst of async RPCs for one nameservice saturates its handler threads and fills the bounded queue: handlers are blocked on a slow or dead backing Namenode while new requests keep arriving; per-ns handler count too low for a hotspot namespace; queue size lowered below the burst depth.","commonSituations":"Heavy MapReduce/Spark/SQL load concentrated on one nameservice behind a Router with default 10 handlers and queue 1000; a Namenode GC pause or outage causing all its async handlers to block; deliberate small queue sizing for load shedding that clients with aggressive retry policies trip over.","solutions":["Treat it as transient: StandbyException is retryable by the standard HDFS client retry policy, so most clients recover without action once backpressure clears.","Raise dedicated handlers for hot namespaces: dfs.federation.router.async.rpc.ns.handler.count=ns1:64,ns2:8 (format nsId:count, comma-separated).","Raise the global fallback dfs.federation.router.async.rpc.handler.count (default 10) if many namespaces are busy, not just one.","Increase dfs.federation.router.async.rpc.queue.size (default 1000) to absorb bursts.","Fix the slow backend: check the Namenode for the named nsid (GC, disk, RPC queue) — handlers free up as soon as the NN responds.","Monitor the router's per-namespace async handler queue size JMX metric (recorded via recordAsyncHandlerQueueSize) and alert before it reaches capacity."],"exampleFix":"<!-- before: defaults, one hot namespace overflows -->\n<property><name>dfs.federation.router.async.rpc.handler.count</name><value>10</value></property>\n\n<!-- after: dedicate handlers to the busy namespace -->\n<property><name>dfs.federation.router.async.rpc.ns.handler.count</name><value>ns1:64</value></property>\n<property><name>dfs.federation.router.async.rpc.queue.size</name><value>5000</value></property>","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// StandbyException is already covered by the default client retry policy;\n// for manual loops, retry with exponential backoff:\nfor (int attempt = 0; attempt < maxAttempts; attempt++) {\n  try {\n    return fs.getFileStatus(path);\n  } catch (RemoteException re) {\n    if (\"org.apache.hadoop.ipc.StandbyException\".equals(re.getClassName())\n        && re.getMessage().contains(\"async handler is busy\")) {\n      Thread.sleep(backoffMs << attempt);\n      continue;\n    }\n    throw re;\n  }\n}","preventionTips":["Size per-namespace handlers for hot namespaces: dfs.federation.router.async.rpc.ns.handler.count=nsX:<count> instead of relying on the default 10.","Alert on the router's async handler queue-size JMX metric well below dfs.federation.router.async.rpc.queue.size so saturation is caught before rejection.","Keep Namenode latency healthy (GC tuning, disk) — handlers only stay busy when the backend is slow.","Do not shrink dfs.federation.router.async.rpc.queue.size without measuring burst depth at peak."],"tags":["hdfs","router-based-federation","async-rpc","backpressure","thread-pool","retryable"],"backgroundTag":"thread-pool-saturation","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}