apache/hadoop · warning · AsyncCallLimitExceededException

Exceeded limit of max asynchronous calls: %d, please configu

Error message

Exceeded limit of max asynchronous calls: %d, please configure %s to adjust it.

What it means

In asynchronous RPC mode, every outstanding call increments a counter; checkAsyncCall() throws AsyncCallLimitExceededException once the in-flight count exceeds ipc.client.async.calls.max (default 100). This is deliberate backpressure so a client cannot queue unbounded async calls whose responses would exhaust memory.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java:1485

  }

  public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
      ConnectionId remoteId, AtomicBoolean fallbackToSimpleAuth,
      AlignmentContext alignmentContext)
      throws IOException {
    return call(rpcKind, rpcRequest, remoteId, RPC.RPC_SERVICE_CLASS_DEFAULT,
        fallbackToSimpleAuth, alignmentContext);
  }

  private void checkAsyncCall() throws IOException {
    if (isAsynchronousMode() && isAsyncCallCheckEabled()) {
      if (asyncCallCounter.incrementAndGet() > maxAsyncCalls) {
        String errMsg = String.format(
            "Exceeded limit of max asynchronous calls: %d, " +
            "please configure %s to adjust it.",
            maxAsyncCalls,
            CommonConfigurationKeys.IPC_CLIENT_ASYNC_CALLS_MAX_KEY);
        throw new AsyncCallLimitExceededException(errMsg);
      }
    }
  }

  Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
                ConnectionId remoteId, int serviceClass,
                AtomicBoolean fallbackToSimpleAuth)
      throws IOException {
    return call(rpcKind, rpcRequest, remoteId, serviceClass,
        fallbackToSimpleAuth, null);
  }

  /**
   * Make a call, passing <code>rpcRequest</code>, to the IPC server defined by
   * <code>remoteId</code>, returning the rpc response.
   *
   * @param rpcKind
   * @param rpcRequest -  contains serialized method and method parameters

View on GitHub (pinned to 2add963021)

Solutions

Throttle the caller: wait for callbacks (releaseAsyncCall decrements the counter) so in-flight count stays under the limit.
Raise ipc.client.async.calls.max in the client Configuration to fit your burst size.
Batch at the API level instead of fanning out single-file async calls (e.g., batched listing, bulk delete).
Catch AsyncCallLimitExceededException and treat it as 'slow down' — back off briefly, then resubmit.

Example fix

// before
client.setAsyncMode(true);
for (Path p : paths) { client.rename(p, target(p)); } // exceeds 100 in-flight calls

// after
<property>
  <name>ipc.client.async.calls.max</name><value>2000</value>
</property>
// and/or throttle in code:
for (Path p : paths) {
  while (client.getAsyncCallCount() >= 2000) { Thread.onSpinWait(); }
  client.rename(p, target(p));
}

Defensive patterns

Strategy: fallback

Validate before calling

// bound your own in-flight async calls before the client rejects them:
int max = conf.getInt("ipc.client.async.calls.max", 100);
while (client.getAsyncCallCount() >= max) {
  // wait for callbacks to drain (or block briefly)
}

Try / catch

try {
  client.callAsync(...);
} catch (AsyncCallLimitExceededException e) {
  // backpressure signal: wait for outstanding callbacks, then resubmit
  waitForCallbacksToDrain();
  client.callAsync(...);
}

Prevention

Throttle fan-out loops to keep in-flight async calls below the limit.
Size ipc.client.async.calls.max to your real burst, not the default 100, if you design for concurrency.
Always consume async results/callbacks so the counter drains.
Batch operations (listing, bulk delete) instead of one async call per item.

When it happens

Trigger: Issuing more than maxAsyncCalls RPC calls via a client put in async mode (client.setAsyncMode(true) / RPC async proxies) before callbacks complete; burst workloads like parallel listing, bulk rename/delete loops, or thousands of async FSNN operations; lowering the max while the workload stays the same.

Common situations: Applications migrating synchronous loops to async Hadoop RPC without throttling;storm-like fan-out from a single JVM; tests that fire many async calls and never wait; noticing that the limit counts all async calls client-wide, not per connection.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/235d63ccf70810ed. Report an issue: GitHub.