apache/beam · error · RuntimeException

Request failed after exhausting retries. Max retries: {maxRe

Error message

Request failed after exhausting retries. Max retries: {maxRetries},

What it means

RetryHandler.execute() retries a request with exponential backoff until the BackOff signals STOP. When retries are exhausted without success, it throws a RuntimeException carrying the max-retry count and the last exception as cause, after logging the failure.

Source

Thrown at sdks/java/ml/inference/remote/src/main/java/org/apache/beam/sdk/ml/inference/remote/RetryHandler.java:102

    int attempt = 0;

    while (true) {
      try {
        return request.call();

      } catch (Exception e) {
        lastException = e;

        if (retryFilter != null && !retryFilter.shouldRetry(e)) {
          LOG.warn("Exception not eligible for retry. Failing immediately.", e);
          throw e;
        }

        long backoffMillis = backoff.nextBackOffMillis();

        if (backoffMillis == BackOff.STOP) {
          LOG.error("Request failed after {} retry attempts.", attempt);
          throw new RuntimeException(
              "Request failed after exhausting retries. " + "Max retries: " + maxRetries + ", ",
              lastException);
        }

        attempt++;
        LOG.warn("Retry request attempt {} failed. Retrying in {} ms", attempt, backoffMillis, e);

        sleeper.sleep(backoffMillis);
      }
    }
  }

  @FunctionalInterface
  public interface RetryableRequest<T> {

    T call() throws Exception;
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Inspect lastException (getCause()) for the underlying network/HTTP failure.
Increase maxRetries and/or the backoff period for transient upstream issues.
Verify the endpoint URL, DNS, and network egress from the Beam workers.
Address server-side causes: rate limits (back off more), service health, capacity.

Example fix

// before
RetryHandler rh = new RetryHandler(options, 2, backoff);
// after
RetryHandler rh = new RetryHandler(options, 10, BackOff.EXPONENTIAL_BACKOFF.withMaxRetries(10).withInitialBackoff(Duration.millis(500)));

Defensive patterns

Strategy: retry

Validate before calling

if (!endpointReachable(url, timeoutMs)) { throw new IllegalStateException("remote inference endpoint unreachable: " + url); }

Try / catch

try { result = retryHandler.executeWithT(() -> callEndpoint()); } catch (RuntimeException e) { if (e.getMessage().contains("exhausting retries")) { /* alert + inspect e.getCause(); consider circuit-breaker */ } throw e; }

Prevention

Size maxRetries/backoff to the endpoint's expected recovery time
Monitor endpoint health and rate limits
Verify worker network egress/DNS before launching
Use idempotent requests so retries are safe

When it happens

Trigger: execute()/executeWithT() called against an endpoint that keeps failing (connection errors, 5xx, timeouts) for more than maxRetries consecutive attempts with the configured backoff.

Common situations: Remote inference endpoint down or misconfigured URL, persistent 429/5xx from an overloaded service, DNS/firewall issues on workers, and maxRetries set too low for a flaky network.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a1827b870dc9e49a. Report an issue: GitHub.