{"record":{"id":"3c5a31edce24f331","repo":"vxcontrol/pentagi","slug":"there-are-requesting-too-many-results-3c5a31","errorCode":null,"errorMessage":"there are requesting too many results","messagePattern":"there are requesting too many results","errorType":"http","errorClass":"Retryable","httpStatus":429,"severity":"warning","filePath":"backend/pkg/tools/searchers/tavily.go","lineNumber":160,"sourceCode":"\tswitch resp.StatusCode {\n\tcase http.StatusOK:\n\t\tvar respBody tavilySearchResult\n\t\tif err := json.NewDecoder(resp.Body).Decode(&respBody); err != nil {\n\t\t\treturn \"\", Fatal(fmt.Errorf(\"failed to decode response body: %v\", err))\n\t\t}\n\t\treturn t.buildTavilyResult(ctx, &respBody), nil\n\tcase http.StatusBadRequest:\n\t\treturn \"\", Fatal(fmt.Errorf(\"request is invalid\"))\n\tcase http.StatusUnauthorized:\n\t\treturn \"\", Fatal(fmt.Errorf(\"API key is wrong\"))\n\tcase http.StatusForbidden:\n\t\treturn \"\", Fatal(fmt.Errorf(\"the endpoint requested is hidden for administrators only\"))\n\tcase http.StatusNotFound:\n\t\treturn \"\", Fatal(fmt.Errorf(\"the specified endpoint could not be found\"))\n\tcase http.StatusMethodNotAllowed:\n\t\treturn \"\", Fatal(fmt.Errorf(\"there need to try to access an endpoint with an invalid method\"))\n\tcase http.StatusTooManyRequests:\n\t\treturn \"\", Retryable(fmt.Errorf(\"there are requesting too many results\"), 0)\n\tcase http.StatusInternalServerError:\n\t\treturn \"\", Retryable(fmt.Errorf(\"there had a problem with our server. try again later\"), 0)\n\tcase http.StatusBadGateway:\n\t\treturn \"\", Retryable(fmt.Errorf(\"there was a problem with the server. Please try again later\"), 0)\n\tcase http.StatusServiceUnavailable:\n\t\treturn \"\", Retryable(fmt.Errorf(\"there are temporarily offline for maintenance. please try again later\"), 0)\n\tcase http.StatusGatewayTimeout:\n\t\treturn \"\", Retryable(fmt.Errorf(\"there are temporarily offline for maintenance. please try again later\"), 0)\n\tdefault:\n\t\treturn \"\", Fatal(fmt.Errorf(\"unexpected status code: %d\", resp.StatusCode))\n\t}\n}\n\nfunc (t *tavily) buildTavilyResult(ctx context.Context, result *tavilySearchResult) string {\n\tvar writer strings.Builder\n\twriter.WriteString(\"# Answer\\n\\n\")\n\twriter.WriteString(result.Answer)\n\twriter.WriteString(\"\\n\\n# Links\\n\\n\")","sourceCodeStart":142,"sourceCodeEnd":178,"githubUrl":"https://github.com/vxcontrol/pentagi/blob/ea665308baaff015b226f308438a68d929d0f29b/backend/pkg/tools/searchers/tavily.go#L142-L178","documentation":"Thrown in tavily.parseHTTPResponse when Tavily answers HTTP 429 Too Many Requests, mapped to \"there are requesting too many results\". The account hit its rate limit (requests per minute or monthly quota). Unlike the 4xx Fatal cases, this is wrapped as Retryable(err, 0), signaling the orchestrator to back off and retry or fall back to another engine.","triggerScenarios":"POST to api.tavily.com/search returns 429: too many concurrent agent flows issuing web_search calls, burst of queries from a single flow, or the free-tier monthly quota is exhausted.","commonSituations":"Multi-agent PentAGI runs firing many searches in parallel on a free Tavily plan (1 req/s style limits); shared API key used by multiple deployments; automated test suites hammering the API; end-of-month quota exhaustion on the free tier.","solutions":["Let the built-in Retryable classification work: ensure the web_search orchestrator retries with backoff and check that a fallback engine is configured for sustained load.","Upgrade the Tavily plan or increase rate limits to match your concurrency.","Add client-side rate limiting/throttling (semaphore or token bucket) around Tavily calls to smooth bursts from parallel agents.","Distribute searches across multiple configured engines (DuckDuckGo, Google, Searxng) instead of pinning everything to Tavily.","Honor the Retry-After response header when backing off instead of retrying immediately."],"exampleFix":"// before: unbounded parallel searches hit 429\nfor _, q := range queries { go t.search(ctx, q, 5) }\n// after: throttle with a semaphore\nsem := make(chan struct{}, 2) // stay under the rate limit\nfor _, q := range queries {\n    sem <- struct{}{}\n    go func(q string) { defer func() { <-sem }(); t.search(ctx, q, 5) }(q)\n}","handlingStrategy":"retry","validationCode":"// client-side budget check before issuing another request\nif !limiter.Allow() { // e.g. golang.org/x/time/rate: rate.Limiter at the plan's QPS\n    return \"\", Retryable(fmt.Errorf(\"local rate limit reached\"), time.Second)\n}","typeGuard":null,"tryCatchPattern":"result, err := searcher.Handle(ctx, req)\nif err != nil {\n    var re *RetryableError\n    if errors.As(err, &re) && strings.Contains(err.Error(), \"too many results\") {\n        select {\n        case <-time.After(backoff(retryN)): // exponential backoff, honor Retry-After if present\n        case <-ctx.Done():\n        }\n        return retry(retryN + 1)\n    }\n    return err\n}","preventionTips":["Add a rate.Limiter around Tavily calls sized to the plan's quota, especially with parallel agent flows.","Upgrade the Tavily plan before scaling concurrency, not after hitting 429s.","Spread load across multiple engines instead of a single default.","Monitor 429 rates and alert before quota exhaustion."],"tags":["rate-limit","http-429","searchers","retryable","backoff"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"ea665308baaff015b226f308438a68d929d0f29b","analyzedAt":"2026-09-01T14:16:31.421Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}