apache/hadoop · error · StandbyException

Router {} is overloaded: Not enough client threads {}/{}

Error message

Router {} is overloaded: Not enough client threads {}/{}

What it means

Error "Router {} is overloaded: Not enough client threads {}/{}" thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java:1692

    final Method m = method.getMethod();
    try {
      List<Future<Object>> futures = null;
      if (timeOutMs > 0) {
        futures = executorService.invokeAll(
            callables, timeOutMs, TimeUnit.MILLISECONDS);
      } else {
        futures = executorService.invokeAll(callables);
      }
      return processFutures(method, m, orderedLocations, futures);
    } catch (RejectedExecutionException e) {
      if (rpcMonitor != null) {
        rpcMonitor.proxyOpFailureClientOverloaded();
      }
      int active = executorService.getActiveCount();
      int total = executorService.getMaximumPoolSize();
      String msg = "Not enough client threads " + active + "/" + total;
      LOG.error(msg);
      throw new StandbyException(
          "Router " + router.getRouterId() + " is overloaded: " + msg);
    } catch (InterruptedException ex) {
      LOG.error("Unexpected error while invoking API: {}", ex.getMessage());
      throw new IOException(
          "Unexpected error while invoking API " + ex.getMessage(), ex);
    } finally {
      releasePermit(CONCURRENT_NS, ugi, method, controller);
    }
  }

  /**
   * Handle all futures during the invokeConcurrent call process.
   *
   * @param <T> The type of the remote location.
   * @param <R> The type of the remote method return.
   * @param method The remote method and parameters to invoke.
   * @param m The method to invoke.
   * @param orderedLocations List of remote locations to call concurrently.

View on GitHub (pinned to 2add963021)

Solutions

1. Increase dfs.federation.router.handler.count (or the per-NS handler counts) to match load.
2. Reduce client concurrency or add more Router instances behind a load balancer.
3. Check fairness settings (dfs.federation.router.fairness.*) that may be rejecting requests.

When it happens

Trigger: Router fairness/overload check rejected the request because available client handler threads are below the required threshold.

Common situations: Traffic spike exhausting the Router's RPC handler pool, or handler count configured too low for the workload.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca05e9eab0f5a3ac. Report an issue: GitHub.