{"record":{"id":"c984fb6c6b34559e","repo":"cilium/cilium","slug":"timed-out-while-waiting-to-be-served-with-d-paral","errorCode":null,"errorMessage":"timed out while waiting to be served with %d parallel requests: %w","messagePattern":"timed out while waiting to be served with (.+?) parallel requests: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/rate/api_limiter.go","lineNumber":699,"sourceCode":"\tif skip {\n\t\tgoto skipRateLimiter\n\t}\n\n\tif parallelRequests > 0 {\n\t\twaitCtx := ctx\n\t\tif l.params.MaxWaitDuration > 0 {\n\t\t\tctx2, cancel := context.WithTimeout(ctx, l.params.MaxWaitDuration)\n\t\t\tdefer cancel()\n\t\t\twaitCtx = ctx2\n\t\t}\n\t\tw := int64(waitSemaphoreResolution / parallelRequests)\n\t\terr2 := l.parallelWaitSemaphore.Acquire(waitCtx, w)\n\t\tif err2 != nil {\n\t\t\tif l.params.Log {\n\t\t\t\tscopedLog.Warn(\"Not processing API request. Wait duration for maximum parallel requests exceeds maximum\", logfields.Error, err2)\n\t\t\t}\n\t\t\treq.outcome = outcomeParallelMaxWait\n\t\t\terr = fmt.Errorf(\"timed out while waiting to be served with %d parallel requests: %w\", parallelRequests, err2)\n\t\t\treturn\n\t\t}\n\t\treq.waitSemaphoreWeight = w\n\t}\n\treq.waitDuration = time.Since(req.scheduleTime)\n\n\tl.mutex.Lock()\n\tif l.limiter != nil {\n\t\tr = l.limiter.Reserve()\n\t\tlimitWaitDuration = r.Delay()\n\n\t\tscopedLog = scopedLog.With(\n\t\t\tlogLimit, fmt.Sprintf(\"%.2f/s\", l.limiter.Limit()),\n\t\t\tlogBurst, l.limiter.Burst(),\n\t\t\tlogWaitDurationLimit, limitWaitDuration,\n\t\t\tlogMaxWaitDurationLimiter, l.params.MaxWaitDuration-req.waitDuration,\n\t\t)\n\t}","sourceCodeStart":681,"sourceCodeEnd":717,"githubUrl":"https://github.com/cilium/cilium/blob/ac7b90affa4baf0642e6685319d56907b3a73a6d/pkg/rate/api_limiter.go#L681-L717","documentation":"In APILimiter.wait, when parallel-requests limiting is active, the request must acquire a slot from parallelWaitSemaphore within MaxWaitDuration (if set). This error is returned when that semaphore acquisition fails — typically a context deadline exceeded because too many parallel requests are already in flight/waiting. It wraps the underlying context error.","triggerScenarios":"Calling Wait when parallelRequests > 0, the parallel wait semaphore is saturated, and the wait exceeds MaxWaitDuration (or the caller's context is cancelled while waiting), e.g. 'max-parallel-requests:4,max-wait-duration:1s' with 5+ concurrent callers.","commonSituations":"Load spikes where concurrent API calls exceed the configured parallelism; downstream slowness causing requests to pile up at the semaphore; MaxWaitDuration set too low for realistic wait times.","solutions":["Increase max-parallel-requests or max-wait-duration in the limiter config to absorb realistic concurrency.","Implement caller-side backoff and retry with jitter on this error instead of immediate retries.","Check for leaks: requests that acquired semaphore slots but never released them (cancels/panics) reduce effective parallelism.","Verify the wrapped error — if it is context.Canceled (not DeadlineExceeded) the caller cancelled, so fix the caller's timeout instead."],"exampleFix":"// before\nNewAPILimiterFromConfig(\"api\", \"max-parallel-requests:4,max-wait-duration:100ms\")\n// after\nNewAPILimiterFromConfig(\"api\", \"max-parallel-requests:16,max-wait-duration:2s\")","handlingStrategy":"retry","validationCode":"// size the config to expected concurrency before running:\n// maxConcurrent <= max-parallel-requests and\n// expectedQueueDelay < max-wait-duration","typeGuard":null,"tryCatchPattern":"var backoff = 100 * time.Millisecond\nfor attempt := 0; attempt < 3; attempt++ {\n    err := limiter.Wait(ctx)\n    if err == nil { break }\n    if errors.Is(err, ErrWaitCancelled) || !strings.Contains(err.Error(), \"timed out while waiting\") {\n        return err\n    }\n    time.Sleep(backoff + time.Duration(rand.Int63n(int64(backoff))))\n    backoff *= 2\n}","preventionTips":["Set max-parallel-requests above your real concurrency","Give max-wait-duration realistic headroom for load spikes","Use jittered exponential backoff, not immediate retries","Audit for semaphore slots leaked by panicking or abandoned requests"],"tags":["go","rate-limiting","timeout","concurrency"],"backgroundTag":"rate-limit-timeout","analyzedSha":"ac7b90affa4baf0642e6685319d56907b3a73a6d","analyzedAt":"2026-08-31T18:27:15.868Z","schemaVersion":2},"datasetVersion":"2026-08-31T19:17:28.585Z"}