apache/dolphinscheduler · warning

Too many request, reach global rate limit, current global qp

Error message

Too many request, reach global rate limit, current global qps is {}

What it means

RateLimitInterceptor.preHandle returns false with HTTP 429 when the global rate limiter has no permits (globalRateLimiter.tryAcquire fails) and trafficConfiguration.isGlobalSwitch() is on. This caps total API server QPS across all users/tokens.

Source

Thrown at dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/RateLimitInterceptor.java:94

                             Object handler) throws ExecutionException {
        // tenant-level rate limit
        if (trafficConfiguration.isTenantSwitch()) {
            final String token = request.getHeader("token");
            if (StringUtils.isNotEmpty(token)) {
                final RateLimiter tenantRateLimiter = tenantRateLimiterCache.get(token);
                if (!tenantRateLimiter.tryAcquire()) {
                    response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
                    log.warn("Too many request, reach tenant token: {} rate limit, current tenant qps is {}",
                            MaskUtils.maskString(token, 6), tenantRateLimiter.getRate());
                    return false;
                }
            }
        }
        // global rate limit
        if (trafficConfiguration.isGlobalSwitch()) {
            if (!globalRateLimiter.tryAcquire()) {
                response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
                log.warn("Too many request, reach global rate limit, current global qps is {}",
                        globalRateLimiter.getRate());
                return false;
            }
        }
        return true;
    }

    public RateLimitInterceptor(ApiConfig.TrafficConfiguration trafficConfiguration) {
        this.trafficConfiguration = trafficConfiguration;
        if (trafficConfiguration.isGlobalSwitch()) {
            this.globalRateLimiter =
                    RateLimiter.create(trafficConfiguration.getMaxGlobalQpsRate(), 1, TimeUnit.SECONDS);
        }
    }

}

View on GitHub (pinned to 02eac45a1b)

Solutions

  1. Increase global-max-qps in the traffic-control configuration to match server capacity.
  2. Add client-side throttling/backoff and spread scheduled API calls over time.
  3. Temporarily set global-switch=false if the limit is misconfigured and blocking production traffic.
  4. Scale out API servers and re-tune limits accordingly.

Example fix

// before
global-switch=true
global-max-qps=10  # whole cluster makes 100 req/s
// after
global-switch=true
global-max-qps=200
Defensive patterns

Strategy: retry

Validate before calling

// central client semaphore sized below global-max-qps
from threading import BoundedSemaphore
import time
sem = BoundedSemaphore(50)  # server global qps = 100
with sem:
    time.sleep(1)
    resp = api.call(...)

Try / catch

for attempt in range(5):
    resp = api.call(...)
    if resp.status_code == 429: time.sleep(min(60, 2 ** attempt)); continue
    break

Prevention

When it happens

Trigger: Aggregate API traffic from all clients exceeds the configured global QPS while global traffic control is enabled; preHandle logs the message and rejects with 429.

Common situations: Cluster-wide bursts (many schedulers, UI auto-refresh, mass workflow imports), load testing, a low global QPS default after enabling traffic control, thundering-herd after an outage.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/c8c4506475e65ceb. Report an issue: GitHub.