Too many request, reach tenant token: {} rate limit, current

Error message

Too many request, reach tenant token: {} rate limit, current tenant qps is {}

What it means

RateLimitInterceptor.preHandle returns false with HTTP 429 when the request's 'token' header exceeds its tenant-level token rate limit (tenantRateLimiter.tryAcquire fails). The log message shows the masked token and configured QPS; no exception is thrown — the request is simply rejected.

Source

Thrown at dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/interceptor/RateLimitInterceptor.java:84

                        tenantQuota = customizeTenantQpsRate.getOrDefault(token,
                                trafficConfiguration.getDefaultTenantQpsRate());
                    }
                    // use tenant default rate limit
                    return RateLimiter.create(tenantQuota, 1, TimeUnit.SECONDS);
                }
            });

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response,
                             Object handler) throws ExecutionException {
        // tenant-level rate limit
        if (trafficConfiguration.isTenantSwitch()) {
            final String token = request.getHeader("token");
            if (StringUtils.isNotEmpty(token)) {
                final RateLimiter tenantRateLimiter = tenantRateLimiterCache.get(token);
                if (!tenantRateLimiter.tryAcquire()) {
                    response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
                    log.warn("Too many request, reach tenant token: {} rate limit, current tenant qps is {}",
                            MaskUtils.maskString(token, 6), tenantRateLimiter.getRate());
                    return false;
                }
            }
        }
        // global rate limit
        if (trafficConfiguration.isGlobalSwitch()) {
            if (!globalRateLimiter.tryAcquire()) {
                response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
                log.warn("Too many request, reach global rate limit, current global qps is {}",
                        globalRateLimiter.getRate());
                return false;
            }
        }
        return true;
    }

    public RateLimitInterceptor(ApiConfig.TrafficConfiguration trafficConfiguration) {

View on GitHub (pinned to 02eac45a1b)

Solutions

Reduce client request rate or add retry with backoff honoring Retry-After/backoff behavior.
Raise the tenant token QPS in the API traffic-control configuration (traffic-control tenant switch/max QPS).
Distribute load across multiple tokens/tenants if the callers are independent.
Disable tenant-level traffic control (isTenantSwitch=false) if rate limiting is not desired.

Example fix

// before
# traffic-control config
tenant-switch=true
tenant-max-qps=1  # scripts burst to 20 req/s -> 429
// after
tenant-switch=true
tenant-max-qps=50

Defensive patterns

Strategy: retry

Validate before calling

// client-side throttle matching tenant QPS
import time
class TenantThrottle:
    def __init__(self, qps): self.interval = 1.0 / qps; self.last = 0
    def wait(self):
        now = time.monotonic(); delta = now - self.last
        if delta < self.interval: time.sleep(self.interval - delta)
        self.last = time.monotonic()

Try / catch

for attempt in range(5):
    resp = api.call(...)
    if resp.status_code == 429:
        time.sleep(2 ** attempt); continue
    break

Prevention

Match client QPS to the tenant token limit
Use exponential backoff on 429 responses
Don't share one token across all automation
Tune tenant-max-qps before bulk operations

When it happens

Trigger: Client sends API requests whose 'token' header maps to a tenant limiter whose permits are exhausted (requests/sec above the configured tenant token QPS while trafficConfiguration.isTenantSwitch() is on).

Common situations: Automated scripts or scheduled jobs bursting API calls above the tenant's configured QPS, load tests against the API server, multiple users sharing one security token, low QPS configured in traffic-control settings.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/2d9dbccfcaa9f1e5. Report an issue: GitHub.