{"record":{"id":"ac5b60ed5434e1cf","repo":"can1357/oh-my-pi","slug":"duckduckgo-blocked-the-request-with-a-bot-detectio","errorCode":null,"errorMessage":"DuckDuckGo blocked the request with a bot-detection challenge. DuckDuckGo throttles automated HTML searches from datacenter/shared-egress IPs; configure a credentialed provider such as Brave, Tavily, Exa, or Kagi for reliable web search.","messagePattern":"DuckDuckGo blocked the request with a bot-detection challenge\\. DuckDuckGo throttles automated HTML searches from datacenter/shared-egress IPs; configure a credentialed provider such as Brave, Tavily, Exa, or Kagi for reliable web search\\.","errorType":"exception","errorClass":"SearchProviderError","httpStatus":429,"severity":"error","filePath":"packages/coding-agent/src/web/search/providers/duckduckgo.ts","lineNumber":325,"sourceCode":"\t\tsignal,\n\t\ttimeoutMs: params.timeoutMs,\n\t\treferer: \"https://html.duckduckgo.com/\",\n\t\tinit: {\n\t\t\tmethod: \"POST\",\n\t\t\tbody: form.toString(),\n\t\t},\n\t\theaders: { \"Content-Type\": \"application/x-www-form-urlencoded\" },\n\t});\n\n\tconst body = page.html;\n\tif (page.status < 200 || page.status >= 300) {\n\t\tconst classified = classifyProviderHttpError(\"duckduckgo\", page.status, body);\n\t\tif (classified) throw classified;\n\t\tthrow new SearchProviderError(\"duckduckgo\", `DuckDuckGo HTML error (${page.status})`, page.status);\n\t}\n\n\tif (isAnomalyResponse(body)) {\n\t\tthrow new SearchProviderError(\n\t\t\t\"duckduckgo\",\n\t\t\t\"DuckDuckGo blocked the request with a bot-detection challenge. DuckDuckGo throttles automated HTML searches from datacenter/shared-egress IPs; configure a credentialed provider such as Brave, Tavily, Exa, or Kagi for reliable web search.\",\n\t\t\t429,\n\t\t);\n\t}\n\n\treturn body;\n}\n\n/** Execute a DuckDuckGo web search via the no-JS HTML frontend. */\nexport async function searchDuckDuckGo(params: SearchParams): Promise<SearchResponse> {\n\tconst numResults = clampNumResults(params.numSearchResults ?? params.limit, DEFAULT_NUM_RESULTS, MAX_NUM_RESULTS);\n\tconst signal = withHardTimeout(params.signal, params.timeoutMs);\n\tconst sources: SearchSource[] = [];\n\tconst seen = new Set<string>();\n\tlet form: URLSearchParams | undefined = createDuckDuckGoForm(params);\n\n\twhile (form && sources.length < numResults) {","sourceCodeStart":307,"sourceCodeEnd":343,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/packages/coding-agent/src/web/search/providers/duckduckgo.ts#L307-L343","documentation":"DuckDuckGo's HTML endpoint sometimes returns an HTTP 200 page containing a bot-detection anomaly page instead of results. isAnomalyResponse detects this body, and callDuckDuckGoHtml throws a SearchProviderError with status 429 explaining that DuckDuckGo throttles automated searches from datacenter/shared-egress IPs and suggesting credentialed providers.","triggerScenarios":"The fetched HTML body passes isAnomalyResponse (challenge/anomaly markers present) even though page.status is 2xx — typical after repeated automated POSTs to html.duckduckgo.com from flagged IPs.","commonSituations":"Running web search from cloud VMs/CI runners with shared egress; high query volume in a short window; VPN/datacenter proxies; containers reusing a blocked IP.","solutions":["Switch to a credentialed provider (Brave, Tavily, Exa, Kagi) — DuckDuckGo offers no official API escape from this throttle.","Change egress IP (residential proxy, different network) or wait for the throttle to expire.","Reduce request rate / add backoff between automated searches.","Catch this error and advance the provider fallback chain."],"exampleFix":"// before\nproviders = [\"duckduckgo\"]; // only DDG, blocked in CI\n// after\nproviders = [\"brave\", \"tavily\", \"duckduckgo\"]; // credentialed providers first","handlingStrategy":"fallback","validationCode":null,"typeGuard":"function isBotChallenge(e: unknown): e is SearchProviderError {\n  return e instanceof SearchProviderError && (e.status === 429 || e.message.includes(\"bot-detection challenge\"));\n}","tryCatchPattern":"try {\n  return await searchDuckDuckGo(params);\n} catch (e) {\n  if (isBotChallenge(e)) return searchTavily(params); // credentialed provider\n  throw e;\n}","preventionTips":["Configure a credentialed provider (Brave, Tavily, Exa, Kagi) as primary in datacenter/CI environments.","Throttle request rate and add jitter between automated DDG searches.","Rotate egress IPs if scraping DDG is unavoidable."],"tags":["rate-limit","bot-detection","duckduckgo","search-provider","http-status"],"backgroundTag":"bot-detection-challenge","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}