What causes "waitCallReturn timed out ${timeout} ${unit}" in apache/hadoop?

Raise the timeout you pass to the wait — it is your deadline and must cover worst-case retry plus failover time.

apache/hadoop · error · TimeoutException

waitCallReturn timed out ${timeout} ${unit}

Error message

waitCallReturn timed out ${timeout} ${unit}

What it means

AsyncCallHandler implements the async proxy mode of RetryInvocationHandler: RPC calls return an AsyncGet handle, and waitCallReturn blocks on the result for a caller-supplied timeout. If the value has not arrived when the budget expires, TimeoutException('waitCallReturn timed out <n> <unit>') is thrown — the underlying call may still complete later, so this is a client-side deadline, not proof of server failure.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java:215

        Preconditions.checkState(set);
      }
    }
  }

  static class AsyncValue<V> {
    private V value;

    synchronized V waitAsyncValue(long timeout, TimeUnit unit)
        throws InterruptedException, TimeoutException {
      if (value != null) {
        return value;
      }
      AsyncGet.Util.wait(this, timeout, unit);
      if (value != null) {
        return value;
      }

      throw new TimeoutException("waitCallReturn timed out "
          + timeout + " " + unit);
    }

    synchronized void set(V v) {
      Preconditions.checkNotNull(v);
      Preconditions.checkState(value == null);
      value = v;
      notify();
    }

    synchronized boolean isDone() {
      return value != null;
    }
  }

  static class AsyncCall extends RetryInvocationHandler.Call {
    private final AsyncCallHandler asyncCallHandler;

View on GitHub (pinned to 2add963021)

Solutions

Raise the timeout you pass to the wait — it is your deadline and must cover worst-case retry plus failover time.
Check server health: slow-RPC logs, RPC queue length, handler saturation — the timeout is usually a symptom of latency, not the disease.
Align the retry policy's cumulative backoff with the wait budget so retries can actually finish inside it.
If you always block anyway, use the plain synchronous proxy, or poll isDone() and overlap other work.

Example fix

// before: budget below retry backoff
V v = asyncCall.get(2, TimeUnit.SECONDS); // TimeoutException under load

// after: budget >= worst-case retry + failover
V v = asyncCall.get(30, TimeUnit.SECONDS);

Defensive patterns

Strategy: retry

Try / catch

try {
  V v = asyncGet.get(timeout, unit);
} catch (java.util.concurrent.TimeoutException te) {
  // call may still complete later: re-wait with a fresh budget,
  // or abandon and re-issue ONLY if the operation is idempotent
}

Prevention

Budget wait timeouts to cover worst-case retry plus failover duration.
Use async mode to overlap waits; do not convert it back to blocking on every call.
Alert on timeout rate and fix server latency instead of shrinking timeouts under load.

When it happens

Trigger: Blocking wait (waitAsyncValue/get) on an async Hadoop RPC proxy with a timeout shorter than end-to-end latency: slow or overloaded server, a retry currently backing off, a failover in progress, GC pauses, or network brownout.

Common situations: Tight client deadlines against a loaded NameNode/Router; retry policies whose cumulative backoff exceeds the wait budget; code adopting the async API but still blocking on every call.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0f0eab1e89a4143a. Report an issue: GitHub.