{"record":{"id":"235d63ccf70810ed","repo":"apache/hadoop","slug":"exceeded-limit-of-max-asynchronous-calls-d-plea","errorCode":null,"errorMessage":"Exceeded limit of max asynchronous calls: %d, please configure %s to adjust it.","messagePattern":"Exceeded limit of max asynchronous calls: (.+?), please configure (.+?) to adjust it\\.","errorType":"exception","errorClass":"AsyncCallLimitExceededException","httpStatus":null,"severity":"warning","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java","lineNumber":1485,"sourceCode":"  }\n\n  public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,\n      ConnectionId remoteId, AtomicBoolean fallbackToSimpleAuth,\n      AlignmentContext alignmentContext)\n      throws IOException {\n    return call(rpcKind, rpcRequest, remoteId, RPC.RPC_SERVICE_CLASS_DEFAULT,\n        fallbackToSimpleAuth, alignmentContext);\n  }\n\n  private void checkAsyncCall() throws IOException {\n    if (isAsynchronousMode() && isAsyncCallCheckEabled()) {\n      if (asyncCallCounter.incrementAndGet() > maxAsyncCalls) {\n        String errMsg = String.format(\n            \"Exceeded limit of max asynchronous calls: %d, \" +\n            \"please configure %s to adjust it.\",\n            maxAsyncCalls,\n            CommonConfigurationKeys.IPC_CLIENT_ASYNC_CALLS_MAX_KEY);\n        throw new AsyncCallLimitExceededException(errMsg);\n      }\n    }\n  }\n\n  Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,\n                ConnectionId remoteId, int serviceClass,\n                AtomicBoolean fallbackToSimpleAuth)\n      throws IOException {\n    return call(rpcKind, rpcRequest, remoteId, serviceClass,\n        fallbackToSimpleAuth, null);\n  }\n\n  /**\n   * Make a call, passing <code>rpcRequest</code>, to the IPC server defined by\n   * <code>remoteId</code>, returning the rpc response.\n   *\n   * @param rpcKind\n   * @param rpcRequest -  contains serialized method and method parameters","sourceCodeStart":1467,"sourceCodeEnd":1503,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1467-L1503","documentation":"In asynchronous RPC mode, every outstanding call increments a counter; checkAsyncCall() throws AsyncCallLimitExceededException once the in-flight count exceeds ipc.client.async.calls.max (default 100). This is deliberate backpressure so a client cannot queue unbounded async calls whose responses would exhaust memory.","triggerScenarios":"Issuing more than maxAsyncCalls RPC calls via a client put in async mode (client.setAsyncMode(true) / RPC async proxies) before callbacks complete; burst workloads like parallel listing, bulk rename/delete loops, or thousands of async FSNN operations; lowering the max while the workload stays the same.","commonSituations":"Applications migrating synchronous loops to async Hadoop RPC without throttling;storm-like fan-out from a single JVM; tests that fire many async calls and never wait; noticing that the limit counts all async calls client-wide, not per connection.","solutions":["Throttle the caller: wait for callbacks (releaseAsyncCall decrements the counter) so in-flight count stays under the limit.","Raise ipc.client.async.calls.max in the client Configuration to fit your burst size.","Batch at the API level instead of fanning out single-file async calls (e.g., batched listing, bulk delete).","Catch AsyncCallLimitExceededException and treat it as 'slow down' — back off briefly, then resubmit."],"exampleFix":"// before\nclient.setAsyncMode(true);\nfor (Path p : paths) { client.rename(p, target(p)); } // exceeds 100 in-flight calls\n\n// after\n<property>\n  <name>ipc.client.async.calls.max</name><value>2000</value>\n</property>\n// and/or throttle in code:\nfor (Path p : paths) {\n  while (client.getAsyncCallCount() >= 2000) { Thread.onSpinWait(); }\n  client.rename(p, target(p));\n}","handlingStrategy":"fallback","validationCode":"// bound your own in-flight async calls before the client rejects them:\nint max = conf.getInt(\"ipc.client.async.calls.max\", 100);\nwhile (client.getAsyncCallCount() >= max) {\n  // wait for callbacks to drain (or block briefly)\n}","typeGuard":null,"tryCatchPattern":"try {\n  client.callAsync(...);\n} catch (AsyncCallLimitExceededException e) {\n  // backpressure signal: wait for outstanding callbacks, then resubmit\n  waitForCallbacksToDrain();\n  client.callAsync(...);\n}","preventionTips":["Throttle fan-out loops to keep in-flight async calls below the limit.","Size ipc.client.async.calls.max to your real burst, not the default 100, if you design for concurrency.","Always consume async results/callbacks so the counter drains.","Batch operations (listing, bulk delete) instead of one async call per item."],"tags":["hadoop","ipc","rpc-client","async","backpressure","throttling","configuration"],"backgroundTag":"async-backpressure-limit","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}