apache/shenyu · error · IllegalStateException

Retry limit exceeded

Error message

Retry limit exceeded

What it means

ExponentialRetryBackoffStrategy builds its Reactor retry spec (initDefaultBackoff, via retrySpec) filtering only IllegalStateException causes. When retries are exhausted it throws IllegalStateException("Retry limit exceeded"), signaling the caller that the operation failed after the maximum number of retries.

Solutions

1. Investigate the underlying IllegalStateException cause (often 'connection prematurely closed') — check backend connection handling and keep-alive settings.
2. Increase the max retry count in the retry strategy configuration.
3. Tune Netty connection-pool settings (maxConnections, pendingAcquire) to avoid pool-related failures.
4. Handle the resulting error in the plugin fallback path so clients get a graceful response.

Example fix

// before
retry: {attempts: 1}
// after
retry: {attempts: 3, backoff: "exponential"}

try {
  return retrySpec.execute(callable);
} catch (IllegalStateException e) {
  if ("Retry limit exceeded".equals(e.getMessage())) {
    return fallbackResponse();
  }
  throw e;
}

When it happens

Trigger: An upstream call repeatedly fails with IllegalStateException (typically 'connection prematurely closed before response' from Reactor Netty) across all retry attempts, so onRetryExhaustedThrow fires.

Common situations: Unstable connections to backend (premature closes), pool exhaustion causing repeated IllegalState failures, backend dropping keep-alive connections, retry max too low for a flaky network.

Related errors

• Request timeout, the maximum number of retry times has been…
• failed to add curator cache.
• doHeartbeat Failed to execute,wait to retry.
• remoteAddress is null
• Max Connections value must be strictly positive

AI-assisted analysis of apache/shenyu@567142e072 (2026-09-12). Data as JSON: /api/errors/4b568f1024e1b0e0. Report an issue: GitHub.

Appendix: source

Thrown at shenyu-plugin/shenyu-plugin-httpclient/src/main/java/org/apache/shenyu/plugin/httpclient/ExponentialRetryBackoffStrategy.java:68

     */
    public Mono<R> execute(final Mono<R> response, final ServerWebExchange exchange, final Duration duration, final int retryTimes) {
        RetryBackoffSpec retrySpec = initDefaultBackoff(retryTimes);
        return response.retryWhen(retrySpec)
                .timeout(duration, Mono.error(() -> new java.util.concurrent.TimeoutException("Response took longer than timeout: " + duration)))
                .doOnError(e -> LOG.error(e.getMessage(), e));
    }

    private RetryBackoffSpec initDefaultBackoff(final int retryTimes) {
        return Retry.backoff(retryTimes, Duration.ofMillis(500))
                .maxBackoff(Duration.ofSeconds(5))
                // Retry only for instantaneous errors
                .transientErrors(true)
                // Add 50% random jitter to the delay time of each retry
                .jitter(0.5d)
                .filter(t -> t instanceof IllegalStateException)
                // When the maximum number of retrys is reached, a specified exception is thrown
                .onRetryExhaustedThrow((retryBackoffSpecErr, retrySignal) -> {
                    throw new IllegalStateException("Retry limit exceeded");
                });
    }
}

View on GitHub (pinned to 567142e072)