apache/beam · error · IOException

Failed to call Rate Limit Service

Error message

Failed to call Rate Limit Service

What it means

fetchTokens() retries the gRPC RateLimitService call up to RPC_RETRY_COUNT times, incrementing rpcErrors on each StatusRuntimeException. When the final attempt also fails, it logs an error and wraps the last StatusRuntimeException in an IOException("Failed to call Rate Limit Service") so the pipeline stage can treat repeated gRPC failures as a retriable I/O problem.

Source

Thrown at sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterFactory.java:183

      }

      // RPC Retry Loop
      RateLimitResponse response = null;
      long startTime = System.currentTimeMillis();
      for (int i = 0; i < RPC_RETRY_COUNT; i++) {
        try {
          response =
              currentStub
                  .withDeadlineAfter(timeoutMillis, java.util.concurrent.TimeUnit.MILLISECONDS)
                  .shouldRateLimit(request);
          long endTime = System.currentTimeMillis();
          rpcLatency.update(endTime - startTime);
          break;
        } catch (StatusRuntimeException e) {
          rpcErrors.inc();
          if (i == RPC_RETRY_COUNT - 1) {
            LOG.error("RateLimitService call failed after {} attempts", RPC_RETRY_COUNT, e);
            throw new IOException("Failed to call Rate Limit Service", e);
          }
          rpcRetries.inc();
          LOG.warn("RateLimitService call failed, retrying", e);
          if (sleeper != null) {
            sleeper.sleep(RPC_RETRY_DELAY_MILLIS);
          }
        }
      }

      if (response == null) {
        throw new IOException("Failed to get response from Rate Limit Service");
      }

      if (response.getOverallCode() == RateLimitResponse.Code.OK) {
        requestsAllowed.inc();
        return true;
      } else if (response.getOverallCode() == RateLimitResponse.Code.OVER_LIMIT) {
        long sleepMillis = 0;

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify the rate-limit service endpoint (host:port) is correct and reachable from Beam workers (network/firewall/DNS).
2. Check the wrapped cause (StatusRuntimeException status) to identify UNAVAILABLE vs other codes and fix the underlying gRPC connectivity issue.
3. Increase RPC_RETRY_COUNT / RPC_RETRY_DELAY_MILLIS for flaky networks, and let Beam retry the DoFn on this IOException.

Example fix

// before
factory = new EnvoyRateLimiterFactory(config.withTarget("ratelimit:9999"), stub); // wrong port
// after
factory = new EnvoyRateLimiterFactory(config.withTarget("ratelimit:8081"), stub); // correct service port

// preflight connectivity check to the rate-limit endpoint before pipeline work
try (Socket s = new Socket()) {
  s.connect(new InetSocketAddress(host, port), 2000); // throws if service unreachable
}

try {
  allowed = factory.allow(ctx, permits);
} catch (IOException e) {
  if (e.getCause() instanceof StatusRuntimeException) {
    Status.Code code = ((StatusRuntimeException) e.getCause()).getStatus().getCode();
    LOG.error("Rate-limit RPC failed after retries: {}", code); // decide fail-open/closed
  }
}

When it happens

Trigger: All RPC_RETRY_COUNT attempts to the Envoy rate-limit service raise StatusRuntimeException — e.g. UNAVAILABLE (service down), DEADLINE_EXCEEDED, or PERMISSION_DENIED — with no successful response in between.

Common situations: Envoy rate-limit service not deployed/unreachable from workers, wrong port or DNS name, network partitions in VPC setups, or the rate-limit service being overloaded.

Related errors

• Failed to get response from Rate Limit Service
• EnvoyRateLimiterFactory requires EnvoyRateLimiterContext
• EnvoyRateLimiterFactory requires EnvoyRateLimiterContext, go
• RateLimitServiceStub is null
• Rate Limit Service returned unknown code:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0b8480daf86f2548. Report an issue: GitHub.