{"record":{"id":"0b8480daf86f2548","repo":"apache/beam","slug":"failed-to-call-rate-limit-service","errorCode":null,"errorMessage":"Failed to call Rate Limit Service","messagePattern":"Failed to call Rate Limit Service","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterFactory.java","lineNumber":183,"sourceCode":"      }\n\n      // RPC Retry Loop\n      RateLimitResponse response = null;\n      long startTime = System.currentTimeMillis();\n      for (int i = 0; i < RPC_RETRY_COUNT; i++) {\n        try {\n          response =\n              currentStub\n                  .withDeadlineAfter(timeoutMillis, java.util.concurrent.TimeUnit.MILLISECONDS)\n                  .shouldRateLimit(request);\n          long endTime = System.currentTimeMillis();\n          rpcLatency.update(endTime - startTime);\n          break;\n        } catch (StatusRuntimeException e) {\n          rpcErrors.inc();\n          if (i == RPC_RETRY_COUNT - 1) {\n            LOG.error(\"RateLimitService call failed after {} attempts\", RPC_RETRY_COUNT, e);\n            throw new IOException(\"Failed to call Rate Limit Service\", e);\n          }\n          rpcRetries.inc();\n          LOG.warn(\"RateLimitService call failed, retrying\", e);\n          if (sleeper != null) {\n            sleeper.sleep(RPC_RETRY_DELAY_MILLIS);\n          }\n        }\n      }\n\n      if (response == null) {\n        throw new IOException(\"Failed to get response from Rate Limit Service\");\n      }\n\n      if (response.getOverallCode() == RateLimitResponse.Code.OK) {\n        requestsAllowed.inc();\n        return true;\n      } else if (response.getOverallCode() == RateLimitResponse.Code.OVER_LIMIT) {\n        long sleepMillis = 0;","sourceCodeStart":165,"sourceCodeEnd":201,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterFactory.java#L165-L201","documentation":"fetchTokens() retries the gRPC RateLimitService call up to RPC_RETRY_COUNT times, incrementing rpcErrors on each StatusRuntimeException. When the final attempt also fails, it logs an error and wraps the last StatusRuntimeException in an IOException(\"Failed to call Rate Limit Service\") so the pipeline stage can treat repeated gRPC failures as a retriable I/O problem.","triggerScenarios":"All RPC_RETRY_COUNT attempts to the Envoy rate-limit service raise StatusRuntimeException — e.g. UNAVAILABLE (service down), DEADLINE_EXCEEDED, or PERMISSION_DENIED — with no successful response in between.","commonSituations":"Envoy rate-limit service not deployed/unreachable from workers, wrong port or DNS name, network partitions in VPC setups, or the rate-limit service being overloaded.","solutions":["Verify the rate-limit service endpoint (host:port) is correct and reachable from Beam workers (network/firewall/DNS).","Check the wrapped cause (StatusRuntimeException status) to identify UNAVAILABLE vs other codes and fix the underlying gRPC connectivity issue.","Increase RPC_RETRY_COUNT / RPC_RETRY_DELAY_MILLIS for flaky networks, and let Beam retry the DoFn on this IOException."],"exampleFix":"// before\nfactory = new EnvoyRateLimiterFactory(config.withTarget(\"ratelimit:9999\"), stub); // wrong port\n// after\nfactory = new EnvoyRateLimiterFactory(config.withTarget(\"ratelimit:8081\"), stub); // correct service port","handlingStrategy":"retry","validationCode":"// preflight connectivity check to the rate-limit endpoint before pipeline work\ntry (Socket s = new Socket()) {\n  s.connect(new InetSocketAddress(host, port), 2000); // throws if service unreachable\n}","typeGuard":null,"tryCatchPattern":"try {\n  allowed = factory.allow(ctx, permits);\n} catch (IOException e) {\n  if (e.getCause() instanceof StatusRuntimeException) {\n    Status.Code code = ((StatusRuntimeException) e.getCause()).getStatus().getCode();\n    LOG.error(\"Rate-limit RPC failed after retries: {}\", code); // decide fail-open/closed\n  }\n}","preventionTips":["Check that the Envoy rate-limit service is deployed and reachable from all workers.","Tune RPC_RETRY_COUNT and RPC_RETRY_DELAY_MILLIS for your network reliability.","Inspect the wrapped StatusRuntimeException status for the root gRPC code.","Let Beam runner-level retry policies handle this retriable IOException."],"tags":["grpc","rate-limiting","envoy","network","retry"],"backgroundTag":"grpc-unavailable","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}