apache/beam · error · IOException

Failed to get response from Rate Limit Service

Error message

Failed to get response from Rate Limit Service

What it means

After the retry loop, fetchTokens() checks whether a RateLimitResponse was obtained; if response is still null (all attempts failed without throwing, or the loop exhausted without assigning a response), it throws IOException("Failed to get response from Rate Limit Service"). This guards against proceeding with a null protobuf response.

Source

Thrown at sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterFactory.java:194

          long endTime = System.currentTimeMillis();
          rpcLatency.update(endTime - startTime);
          break;
        } catch (StatusRuntimeException e) {
          rpcErrors.inc();
          if (i == RPC_RETRY_COUNT - 1) {
            LOG.error("RateLimitService call failed after {} attempts", RPC_RETRY_COUNT, e);
            throw new IOException("Failed to call Rate Limit Service", e);
          }
          rpcRetries.inc();
          LOG.warn("RateLimitService call failed, retrying", e);
          if (sleeper != null) {
            sleeper.sleep(RPC_RETRY_DELAY_MILLIS);
          }
        }
      }

      if (response == null) {
        throw new IOException("Failed to get response from Rate Limit Service");
      }

      if (response.getOverallCode() == RateLimitResponse.Code.OK) {
        requestsAllowed.inc();
        return true;
      } else if (response.getOverallCode() == RateLimitResponse.Code.OVER_LIMIT) {
        long sleepMillis = 0;
        for (RateLimitResponse.DescriptorStatus status : response.getStatusesList()) {
          if (status.getCode() == RateLimitResponse.Code.OVER_LIMIT
              && status.hasDurationUntilReset()) {
            long durationMillis =
                status.getDurationUntilReset().getSeconds() * 1000
                    + status.getDurationUntilReset().getNanos() / 1_000_000;
            if (durationMillis > sleepMillis) {
              sleepMillis = durationMillis;
            }
          }
        }

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify gRPC deadlines are configured so failures surface as StatusRuntimeException (handled by the retry path) instead of null responses.
2. Increase RPC_RETRY_COUNT or RPC_RETRY_DELAY_MILLIS to tolerate transient network hiccups.
3. Confirm the Envoy rate-limit service is healthy; treat this IOException as a service-availability signal in pipeline retry logic.

Example fix

// before
stub = RateLimitServiceGrpc.newBlockingStub(channel); // no deadline -> indefinite/null behavior
// after
stub = RateLimitServiceGrpc.newBlockingStub(channel).withDeadlineAfter(500, TimeUnit.MILLISECONDS);

// ensure the endpoint responds before relying on allow()
RateLimitResponse probe = stub.withDeadlineAfter(1, TimeUnit.SECONDS).shouldRateLimit(request);
Objects.requireNonNull(probe, "rate-limit service returned null response");

try {
  allowed = factory.allow(ctx, permits);
} catch (IOException e) {
  // null response: service is unhealthy — back off and retry, or fail open/closed by policy
}

When it happens

Trigger: The retry loop exits without ever receiving a RateLimitResponse and without the StatusRuntimeException path firing — e.g. the stub returns null on the last attempt after repeated failures, or the loop structure leaves response unassigned on exhaustion.

Common situations: Intermittent gRPC failures that align exactly with the retry budget, proxies/load balancers dropping the RPC silently, or timeout configurations that surface as null responses rather than exceptions.

Related errors

• Failed to call Rate Limit Service
• EnvoyRateLimiterFactory requires EnvoyRateLimiterContext
• EnvoyRateLimiterFactory requires EnvoyRateLimiterContext, go
• RateLimitServiceStub is null
• Rate Limit Service returned unknown code:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/57b2a1494d2f87ca. Report an issue: GitHub.