cilium/cilium · error

timed out while waiting to be served with %d parallel reques

Error message

timed out while waiting to be served with %d parallel requests: %w

What it means

In APILimiter.wait, when parallel-requests limiting is active, the request must acquire a slot from parallelWaitSemaphore within MaxWaitDuration (if set). This error is returned when that semaphore acquisition fails — typically a context deadline exceeded because too many parallel requests are already in flight/waiting. It wraps the underlying context error.

Source

Thrown at pkg/rate/api_limiter.go:699

	if skip {
		goto skipRateLimiter
	}

	if parallelRequests > 0 {
		waitCtx := ctx
		if l.params.MaxWaitDuration > 0 {
			ctx2, cancel := context.WithTimeout(ctx, l.params.MaxWaitDuration)
			defer cancel()
			waitCtx = ctx2
		}
		w := int64(waitSemaphoreResolution / parallelRequests)
		err2 := l.parallelWaitSemaphore.Acquire(waitCtx, w)
		if err2 != nil {
			if l.params.Log {
				scopedLog.Warn("Not processing API request. Wait duration for maximum parallel requests exceeds maximum", logfields.Error, err2)
			}
			req.outcome = outcomeParallelMaxWait
			err = fmt.Errorf("timed out while waiting to be served with %d parallel requests: %w", parallelRequests, err2)
			return
		}
		req.waitSemaphoreWeight = w
	}
	req.waitDuration = time.Since(req.scheduleTime)

	l.mutex.Lock()
	if l.limiter != nil {
		r = l.limiter.Reserve()
		limitWaitDuration = r.Delay()

		scopedLog = scopedLog.With(
			logLimit, fmt.Sprintf("%.2f/s", l.limiter.Limit()),
			logBurst, l.limiter.Burst(),
			logWaitDurationLimit, limitWaitDuration,
			logMaxWaitDurationLimiter, l.params.MaxWaitDuration-req.waitDuration,
		)
	}

View on GitHub (pinned to ac7b90affa)

Solutions

1. Increase max-parallel-requests or max-wait-duration in the limiter config to absorb realistic concurrency.
2. Implement caller-side backoff and retry with jitter on this error instead of immediate retries.
3. Check for leaks: requests that acquired semaphore slots but never released them (cancels/panics) reduce effective parallelism.
4. Verify the wrapped error — if it is context.Canceled (not DeadlineExceeded) the caller cancelled, so fix the caller's timeout instead.

Example fix

// before
NewAPILimiterFromConfig("api", "max-parallel-requests:4,max-wait-duration:100ms")
// after
NewAPILimiterFromConfig("api", "max-parallel-requests:16,max-wait-duration:2s")

// size the config to expected concurrency before running:
// maxConcurrent <= max-parallel-requests and
// expectedQueueDelay < max-wait-duration

var backoff = 100 * time.Millisecond
for attempt := 0; attempt < 3; attempt++ {
    err := limiter.Wait(ctx)
    if err == nil { break }
    if errors.Is(err, ErrWaitCancelled) || !strings.Contains(err.Error(), "timed out while waiting") {
        return err
    }
    time.Sleep(backoff + time.Duration(rand.Int63n(int64(backoff))))
    backoff *= 2
}

When it happens

Trigger: Calling Wait when parallelRequests > 0, the parallel wait semaphore is saturated, and the wait exceeds MaxWaitDuration (or the caller's context is cancelled while waiting), e.g. 'max-parallel-requests:4,max-wait-duration:1s' with 5+ concurrent callers.

Common situations: Load spikes where concurrent API calls exceed the configured parallelism; downstream slowness causing requests to pile up at the semaphore; MaxWaitDuration set too low for realistic wait times.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• failed to acquire lock: %w
• exec timeout
• Cilium API client timeout exceeded
• timed out waiting for cluster configuration watcher to be st
• Waiting on %s: %w

AI-assisted analysis of cilium/cilium@ac7b90affa (2026-08-31). Data as JSON: /api/errors/c984fb6c6b34559e. Report an issue: GitHub.