{"record":{"id":"0f0eab1e89a4143a","repo":"apache/hadoop","slug":"waitcallreturn-timed-out-timeout-unit","errorCode":null,"errorMessage":"waitCallReturn timed out ${timeout} ${unit}","messagePattern":"waitCallReturn timed out (.+?) (.+?)","errorType":"exception","errorClass":"TimeoutException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java","lineNumber":215,"sourceCode":"        Preconditions.checkState(set);\n      }\n    }\n  }\n\n  static class AsyncValue<V> {\n    private V value;\n\n    synchronized V waitAsyncValue(long timeout, TimeUnit unit)\n        throws InterruptedException, TimeoutException {\n      if (value != null) {\n        return value;\n      }\n      AsyncGet.Util.wait(this, timeout, unit);\n      if (value != null) {\n        return value;\n      }\n\n      throw new TimeoutException(\"waitCallReturn timed out \"\n          + timeout + \" \" + unit);\n    }\n\n    synchronized void set(V v) {\n      Preconditions.checkNotNull(v);\n      Preconditions.checkState(value == null);\n      value = v;\n      notify();\n    }\n\n    synchronized boolean isDone() {\n      return value != null;\n    }\n  }\n\n  static class AsyncCall extends RetryInvocationHandler.Call {\n    private final AsyncCallHandler asyncCallHandler;\n","sourceCodeStart":197,"sourceCodeEnd":233,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java#L197-L233","documentation":"AsyncCallHandler implements the async proxy mode of RetryInvocationHandler: RPC calls return an AsyncGet handle, and waitCallReturn blocks on the result for a caller-supplied timeout. If the value has not arrived when the budget expires, TimeoutException('waitCallReturn timed out <n> <unit>') is thrown — the underlying call may still complete later, so this is a client-side deadline, not proof of server failure.","triggerScenarios":"Blocking wait (waitAsyncValue/get) on an async Hadoop RPC proxy with a timeout shorter than end-to-end latency: slow or overloaded server, a retry currently backing off, a failover in progress, GC pauses, or network brownout.","commonSituations":"Tight client deadlines against a loaded NameNode/Router; retry policies whose cumulative backoff exceeds the wait budget; code adopting the async API but still blocking on every call.","solutions":["Raise the timeout you pass to the wait — it is your deadline and must cover worst-case retry plus failover time.","Check server health: slow-RPC logs, RPC queue length, handler saturation — the timeout is usually a symptom of latency, not the disease.","Align the retry policy's cumulative backoff with the wait budget so retries can actually finish inside it.","If you always block anyway, use the plain synchronous proxy, or poll isDone() and overlap other work."],"exampleFix":"// before: budget below retry backoff\nV v = asyncCall.get(2, TimeUnit.SECONDS); // TimeoutException under load\n\n// after: budget >= worst-case retry + failover\nV v = asyncCall.get(30, TimeUnit.SECONDS);","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  V v = asyncGet.get(timeout, unit);\n} catch (java.util.concurrent.TimeoutException te) {\n  // call may still complete later: re-wait with a fresh budget,\n  // or abandon and re-issue ONLY if the operation is idempotent\n}","preventionTips":["Budget wait timeouts to cover worst-case retry plus failover duration.","Use async mode to overlap waits; do not convert it back to blocking on every call.","Alert on timeout rate and fix server latency instead of shrinking timeouts under load."],"tags":["async-rpc","timeout","retry","hadoop-common"],"backgroundTag":"operation-timeout","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}