{"record":{"id":"88060cef22e4aff5","repo":"Tencent/WeKnora","slug":"duckduckgo-html-returned-status-d","errorCode":null,"errorMessage":"duckduckgo HTML returned status %d","messagePattern":"duckduckgo HTML returned status (.+?)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/infrastructure/web_search/duckduckgo.go","lineNumber":100,"sourceCode":"\treq.Header.Set(\n\t\t\"User-Agent\",\n\t\t\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\",\n\t)\n\n\tcurlCommand := fmt.Sprintf(\n\t\t\"curl -X GET '%s' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'\",\n\t\treq.URL.String(),\n\t)\n\tlogger.Infof(ctx, \"Curl of request: %s\", secutils.SanitizeForLog(curlCommand))\n\n\tresp, err := p.client.Do(req)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to perform request: %w\", err)\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted {\n\t\treturn nil, fmt.Errorf(\"duckduckgo HTML returned status %d\", resp.StatusCode)\n\t}\n\n\tdoc, err := goquery.NewDocumentFromReader(resp.Body)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to parse HTML: %w\", err)\n\t}\n\n\tresults := make([]*types.WebSearchResult, 0, maxResults)\n\tdoc.Find(\".web-result\").Each(func(i int, s *goquery.Selection) {\n\t\tif len(results) >= maxResults {\n\t\t\treturn\n\t\t}\n\t\ttitleNode := s.Find(\".result__a\")\n\t\ttitle := strings.TrimSpace(titleNode.Text())\n\t\tvar link string\n\t\tif href, exists := titleNode.Attr(\"href\"); exists {\n\t\t\tlink = cleanDDGURL(href)\n\t\t}","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/infrastructure/web_search/duckduckgo.go#L82-L118","documentation":"DuckDuckGo's HTML endpoint returned a status other than 200 or 202, so searchHTML rejects it with this formatted status code error. It means the scrape request reached DuckDuckGo but was not served the normal results page.","triggerScenarios":"DuckDuckGo responds 202 (anomaly/bot challenge), 403, 429, or 5xx to the HTML search request issued via Search.","commonSituations":"Rate limiting from too many rapid scrapes; datacenter IPs flagged for bot challenges; missing/expired cookies or stale User-Agent; regional blocks.","solutions":["Check the status code in the message: 202/403 usually means bot detection — slow down, rotate IPs/proxies, keep headers consistent.","429 -> back off significantly and retry later.","5xx -> retry with backoff; may be transient DuckDuckGo outage.","Rely on the Instant Answer API fallback or switch providers for programmatic access."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"func isDDGStatusErr(err error) (int, bool) {\n    if err == nil { return 0, false }\n    var status int\n    if _, scanErr := fmt.Sscanf(err.Error(), \"duckduckgo HTML returned status %d\", &status); scanErr == nil {\n        return status, true\n    }\n    return 0, false\n}","tryCatchPattern":"results, err := ddg.Search(ctx, query, 10, false)\nif err != nil {\n    if strings.Contains(err.Error(), \"duckduckgo HTML returned status\") {\n        // blocked or throttled: try API path or another provider after backoff\n        return altProvider.Search(ctx, query, 10, false)\n    }\n    return nil, err\n}","preventionTips":["Throttle scrape frequency well below what triggers anomaly detection.","Keep User-Agent and headers stable and realistic across requests.","Treat 202 responses as challenges, not success, in monitoring.","Prefer official APIs over HTML scraping for production workloads."],"tags":["http","scraping","rate-limit","web-search","go"],"backgroundTag":"http-non-200-response","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}