{"record":{"id":"57b2a1494d2f87ca","repo":"apache/beam","slug":"failed-to-get-response-from-rate-limit-service","errorCode":null,"errorMessage":"Failed to get response from Rate Limit Service","messagePattern":"Failed to get response from Rate Limit Service","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterFactory.java","lineNumber":194,"sourceCode":"          long endTime = System.currentTimeMillis();\n          rpcLatency.update(endTime - startTime);\n          break;\n        } catch (StatusRuntimeException e) {\n          rpcErrors.inc();\n          if (i == RPC_RETRY_COUNT - 1) {\n            LOG.error(\"RateLimitService call failed after {} attempts\", RPC_RETRY_COUNT, e);\n            throw new IOException(\"Failed to call Rate Limit Service\", e);\n          }\n          rpcRetries.inc();\n          LOG.warn(\"RateLimitService call failed, retrying\", e);\n          if (sleeper != null) {\n            sleeper.sleep(RPC_RETRY_DELAY_MILLIS);\n          }\n        }\n      }\n\n      if (response == null) {\n        throw new IOException(\"Failed to get response from Rate Limit Service\");\n      }\n\n      if (response.getOverallCode() == RateLimitResponse.Code.OK) {\n        requestsAllowed.inc();\n        return true;\n      } else if (response.getOverallCode() == RateLimitResponse.Code.OVER_LIMIT) {\n        long sleepMillis = 0;\n        for (RateLimitResponse.DescriptorStatus status : response.getStatusesList()) {\n          if (status.getCode() == RateLimitResponse.Code.OVER_LIMIT\n              && status.hasDurationUntilReset()) {\n            long durationMillis =\n                status.getDurationUntilReset().getSeconds() * 1000\n                    + status.getDurationUntilReset().getNanos() / 1_000_000;\n            if (durationMillis > sleepMillis) {\n              sleepMillis = durationMillis;\n            }\n          }\n        }","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/components/src/main/java/org/apache/beam/sdk/io/components/ratelimiter/EnvoyRateLimiterFactory.java#L176-L212","documentation":"After the retry loop, fetchTokens() checks whether a RateLimitResponse was obtained; if response is still null (all attempts failed without throwing, or the loop exhausted without assigning a response), it throws IOException(\"Failed to get response from Rate Limit Service\"). This guards against proceeding with a null protobuf response.","triggerScenarios":"The retry loop exits without ever receiving a RateLimitResponse and without the StatusRuntimeException path firing — e.g. the stub returns null on the last attempt after repeated failures, or the loop structure leaves response unassigned on exhaustion.","commonSituations":"Intermittent gRPC failures that align exactly with the retry budget, proxies/load balancers dropping the RPC silently, or timeout configurations that surface as null responses rather than exceptions.","solutions":["Verify gRPC deadlines are configured so failures surface as StatusRuntimeException (handled by the retry path) instead of null responses.","Increase RPC_RETRY_COUNT or RPC_RETRY_DELAY_MILLIS to tolerate transient network hiccups.","Confirm the Envoy rate-limit service is healthy; treat this IOException as a service-availability signal in pipeline retry logic."],"exampleFix":"// before\nstub = RateLimitServiceGrpc.newBlockingStub(channel); // no deadline -> indefinite/null behavior\n// after\nstub = RateLimitServiceGrpc.newBlockingStub(channel).withDeadlineAfter(500, TimeUnit.MILLISECONDS);","handlingStrategy":"retry","validationCode":"// ensure the endpoint responds before relying on allow()\nRateLimitResponse probe = stub.withDeadlineAfter(1, TimeUnit.SECONDS).shouldRateLimit(request);\nObjects.requireNonNull(probe, \"rate-limit service returned null response\");","typeGuard":null,"tryCatchPattern":"try {\n  allowed = factory.allow(ctx, permits);\n} catch (IOException e) {\n  // null response: service is unhealthy — back off and retry, or fail open/closed by policy\n}","preventionTips":["Set explicit gRPC deadlines so failures surface as exceptions handled by the retry loop.","Monitor rate-limit service health/uptime from workers.","Treat repeated occurrences as a service availability incident, not a client bug."],"tags":["grpc","rate-limiting","envoy","network"],"backgroundTag":"empty-response-body","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}