{"record":{"id":"c8c4506475e65ceb","repo":"apache/dolphinscheduler","slug":"too-many-request-reach-global-rate-limit-current","errorCode":null,"errorMessage":"Too many request, reach global rate limit, current global qps is {}","messagePattern":"Too many request, reach global rate limit, current global qps is (.+?)","errorType":"http","errorClass":null,"httpStatus":429,"severity":"warning","filePath":"dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/RateLimitInterceptor.java","lineNumber":94,"sourceCode":"                             Object handler) throws ExecutionException {\n        // tenant-level rate limit\n        if (trafficConfiguration.isTenantSwitch()) {\n            final String token = request.getHeader(\"token\");\n            if (StringUtils.isNotEmpty(token)) {\n                final RateLimiter tenantRateLimiter = tenantRateLimiterCache.get(token);\n                if (!tenantRateLimiter.tryAcquire()) {\n                    response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());\n                    log.warn(\"Too many request, reach tenant token: {} rate limit, current tenant qps is {}\",\n                            MaskUtils.maskString(token, 6), tenantRateLimiter.getRate());\n                    return false;\n                }\n            }\n        }\n        // global rate limit\n        if (trafficConfiguration.isGlobalSwitch()) {\n            if (!globalRateLimiter.tryAcquire()) {\n                response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());\n                log.warn(\"Too many request, reach global rate limit, current global qps is {}\",\n                        globalRateLimiter.getRate());\n                return false;\n            }\n        }\n        return true;\n    }\n\n    public RateLimitInterceptor(ApiConfig.TrafficConfiguration trafficConfiguration) {\n        this.trafficConfiguration = trafficConfiguration;\n        if (trafficConfiguration.isGlobalSwitch()) {\n            this.globalRateLimiter =\n                    RateLimiter.create(trafficConfiguration.getMaxGlobalQpsRate(), 1, TimeUnit.SECONDS);\n        }\n    }\n\n}\n","sourceCodeStart":76,"sourceCodeEnd":111,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/RateLimitInterceptor.java#L76-L111","documentation":"RateLimitInterceptor.preHandle returns false with HTTP 429 when the global rate limiter has no permits (globalRateLimiter.tryAcquire fails) and trafficConfiguration.isGlobalSwitch() is on. This caps total API server QPS across all users/tokens.","triggerScenarios":"Aggregate API traffic from all clients exceeds the configured global QPS while global traffic control is enabled; preHandle logs the message and rejects with 429.","commonSituations":"Cluster-wide bursts (many schedulers, UI auto-refresh, mass workflow imports), load testing, a low global QPS default after enabling traffic control, thundering-herd after an outage.","solutions":["Increase global-max-qps in the traffic-control configuration to match server capacity.","Add client-side throttling/backoff and spread scheduled API calls over time.","Temporarily set global-switch=false if the limit is misconfigured and blocking production traffic.","Scale out API servers and re-tune limits accordingly."],"exampleFix":"// before\nglobal-switch=true\nglobal-max-qps=10  # whole cluster makes 100 req/s\n// after\nglobal-switch=true\nglobal-max-qps=200","handlingStrategy":"retry","validationCode":"// central client semaphore sized below global-max-qps\nfrom threading import BoundedSemaphore\nimport time\nsem = BoundedSemaphore(50)  # server global qps = 100\nwith sem:\n    time.sleep(1)\n    resp = api.call(...)","typeGuard":null,"tryCatchPattern":"for attempt in range(5):\n    resp = api.call(...)\n    if resp.status_code == 429: time.sleep(min(60, 2 ** attempt)); continue\n    break","preventionTips":["Set global-max-qps below total server capacity","Spread batch jobs across time windows","Add backoff in all API automation","Alert on rising 429 counts at the load balancer"],"tags":["rate-limit","http-429","api","global"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}