apache/hadoop · warning · StandbyException

Namespace '{nsid}' async handler is busy.

Error message

Namespace '{nsid}' async handler is busy.

What it means

A StandbyException raised when the per-nameservice asynchronous handler thread pool rejects a task. RouterRpcServer creates one bounded ThreadPoolExecutor per nameservice (handlers = dfs.federation.router.async.rpc.handler.count, default 10, or per-ns dfs.federation.router.async.rpc.ns.handler.count; queue = LinkedBlockingQueue of dfs.federation.router.async.rpc.queue.size, default 1000). When the queue is full, asyncApplyUseExecutor completes exceptionally with RejectedExecutionException, and asyncCatch converts it into StandbyException so clients treat the Router like a standby Namenode and retry. This is intentional backpressure to prevent unbounded memory growth.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/async/RouterAsyncRpcClient.java:199

    asyncApplyUseExecutor((AsyncApplyFunction<Object, Object>) o -> {
      if (LOG.isDebugEnabled()) {
        LOG.debug("Async invoke method : {}, {}, {}, {}", method.getName(), useObserver, namenodes,
            params);
      }
      threadLocalContext.transfer();
      RouterRpcFairnessPolicyController controller = getRouterRpcFairnessPolicyController();
      acquirePermit(nsid, ugi, method.getName(), controller);
      invokeMethodAsync(ugi, (List<FederationNamenodeContext>) namenodes,
          useObserver, protocol, method, params);
      asyncFinally(object -> {
        releasePermit(nsid, ugi, method, controller);
        return object;
      });
    }, router.getRpcServer().getAsyncExecutorForNamespace(nsid));

    // Catch the RejectedExecutionException and convert it to StandbyException
    asyncCatch((ret, e) -> {
      throw new StandbyException("Namespace '" + nsid + "' async handler is busy.");
    }, RejectedExecutionException.class);
    return null;
  }

  /**
   * Asynchronously invokes a method on the specified NameNodes for a given user and operation.
   * This method is responsible for the actual execution of the remote method call on the
   * NameNodes in a non-blocking manner, allowing for concurrent processing.
   *
   * <p>In case of exceptions, the method includes logic to handle retries, failover to standby
   * NameNodes, and proper exception handling to ensure that the calling code can respond
   * appropriately to different error conditions.
   *
   * @param ugi The user information under which the method is to be invoked.
   * @param namenodes The list of NameNode contexts on which the method will be invoked.
   * @param useObserver Whether to use an observer node for the invocation if available.
   * @param protocol The protocol class defining the method to be invoked.
   * @param method The method to be invoked on the NameNodes.

View on GitHub (pinned to 2add963021)

Solutions

1. Treat it as transient: StandbyException is retryable by the standard HDFS client retry policy, so most clients recover without action once backpressure clears.
2. Raise dedicated handlers for hot namespaces: dfs.federation.router.async.rpc.ns.handler.count=ns1:64,ns2:8 (format nsId:count, comma-separated).
3. Raise the global fallback dfs.federation.router.async.rpc.handler.count (default 10) if many namespaces are busy, not just one.
4. Increase dfs.federation.router.async.rpc.queue.size (default 1000) to absorb bursts.
5. Fix the slow backend: check the Namenode for the named nsid (GC, disk, RPC queue) — handlers free up as soon as the NN responds.
6. Monitor the router's per-namespace async handler queue size JMX metric (recorded via recordAsyncHandlerQueueSize) and alert before it reaches capacity.

Example fix

<!-- before: defaults, one hot namespace overflows -->
<property><name>dfs.federation.router.async.rpc.handler.count</name><value>10</value></property>

<!-- after: dedicate handlers to the busy namespace -->
<property><name>dfs.federation.router.async.rpc.ns.handler.count</name><value>ns1:64</value></property>
<property><name>dfs.federation.router.async.rpc.queue.size</name><value>5000</value></property>

// StandbyException is already covered by the default client retry policy;
// for manual loops, retry with exponential backoff:
for (int attempt = 0; attempt < maxAttempts; attempt++) {
  try {
    return fs.getFileStatus(path);
  } catch (RemoteException re) {
    if ("org.apache.hadoop.ipc.StandbyException".equals(re.getClassName())
        && re.getMessage().contains("async handler is busy")) {
      Thread.sleep(backoffMs << attempt);
      continue;
    }
    throw re;
  }
}

When it happens

Trigger: A burst of async RPCs for one nameservice saturates its handler threads and fills the bounded queue: handlers are blocked on a slow or dead backing Namenode while new requests keep arriving; per-ns handler count too low for a hotspot namespace; queue size lowered below the burst depth.

Common situations: Heavy MapReduce/Spark/SQL load concentrated on one nameservice behind a Router with default 10 handlers and queue 1000; a Namenode GC pause or outage causing all its async handlers to block; deliberate small queue sizing for load shedding that clients with aggressive retry policies trip over.

Related errors

• No namenodes to invoke {methodName} with params {params} fro
• Zero blocklocations for {}. Name node is in safe mode. {} Na
• Federated metrics is not initialized
• Cannot serialize field {} into JSON
• The quota system is disabled in Router.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c3b0345ae1f69e20. Report an issue: GitHub.