{"record":{"id":"2d9dbccfcaa9f1e5","repo":"apache/dolphinscheduler","slug":"too-many-request-reach-tenant-token-rate-limi","errorCode":null,"errorMessage":"Too many request, reach tenant token: {} rate limit, current tenant qps is {}","messagePattern":"Too many request, reach tenant token: (.+?) rate limit, current tenant qps is (.+?)","errorType":"http","errorClass":null,"httpStatus":429,"severity":"warning","filePath":"dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/RateLimitInterceptor.java","lineNumber":84,"sourceCode":"                        tenantQuota = customizeTenantQpsRate.getOrDefault(token,\n                                trafficConfiguration.getDefaultTenantQpsRate());\n                    }\n                    // use tenant default rate limit\n                    return RateLimiter.create(tenantQuota, 1, TimeUnit.SECONDS);\n                }\n            });\n\n    @Override\n    public boolean preHandle(HttpServletRequest request, HttpServletResponse response,\n                             Object handler) throws ExecutionException {\n        // tenant-level rate limit\n        if (trafficConfiguration.isTenantSwitch()) {\n            final String token = request.getHeader(\"token\");\n            if (StringUtils.isNotEmpty(token)) {\n                final RateLimiter tenantRateLimiter = tenantRateLimiterCache.get(token);\n                if (!tenantRateLimiter.tryAcquire()) {\n                    response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());\n                    log.warn(\"Too many request, reach tenant token: {} rate limit, current tenant qps is {}\",\n                            MaskUtils.maskString(token, 6), tenantRateLimiter.getRate());\n                    return false;\n                }\n            }\n        }\n        // global rate limit\n        if (trafficConfiguration.isGlobalSwitch()) {\n            if (!globalRateLimiter.tryAcquire()) {\n                response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());\n                log.warn(\"Too many request, reach global rate limit, current global qps is {}\",\n                        globalRateLimiter.getRate());\n                return false;\n            }\n        }\n        return true;\n    }\n\n    public RateLimitInterceptor(ApiConfig.TrafficConfiguration trafficConfiguration) {","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/RateLimitInterceptor.java#L66-L102","documentation":"RateLimitInterceptor.preHandle returns false with HTTP 429 when the request's 'token' header exceeds its tenant-level token rate limit (tenantRateLimiter.tryAcquire fails). The log message shows the masked token and configured QPS; no exception is thrown — the request is simply rejected.","triggerScenarios":"Client sends API requests whose 'token' header maps to a tenant limiter whose permits are exhausted (requests/sec above the configured tenant token QPS while trafficConfiguration.isTenantSwitch() is on).","commonSituations":"Automated scripts or scheduled jobs bursting API calls above the tenant's configured QPS, load tests against the API server, multiple users sharing one security token, low QPS configured in traffic-control settings.","solutions":["Reduce client request rate or add retry with backoff honoring Retry-After/backoff behavior.","Raise the tenant token QPS in the API traffic-control configuration (traffic-control tenant switch/max QPS).","Distribute load across multiple tokens/tenants if the callers are independent.","Disable tenant-level traffic control (isTenantSwitch=false) if rate limiting is not desired."],"exampleFix":"// before\n# traffic-control config\ntenant-switch=true\ntenant-max-qps=1  # scripts burst to 20 req/s -> 429\n// after\ntenant-switch=true\ntenant-max-qps=50","handlingStrategy":"retry","validationCode":"// client-side throttle matching tenant QPS\nimport time\nclass TenantThrottle:\n    def __init__(self, qps): self.interval = 1.0 / qps; self.last = 0\n    def wait(self):\n        now = time.monotonic(); delta = now - self.last\n        if delta < self.interval: time.sleep(self.interval - delta)\n        self.last = time.monotonic()","typeGuard":null,"tryCatchPattern":"for attempt in range(5):\n    resp = api.call(...)\n    if resp.status_code == 429:\n        time.sleep(2 ** attempt); continue\n    break","preventionTips":["Match client QPS to the tenant token limit","Use exponential backoff on 429 responses","Don't share one token across all automation","Tune tenant-max-qps before bulk operations"],"tags":["rate-limit","http-429","api","interceptor"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}