{"record":{"id":"4402b3a7d9138808","repo":"Tencent/WeKnora","slug":"duckduckgo-html-search-failed-w","errorCode":null,"errorMessage":"duckduckgo HTML search failed: %w","messagePattern":"duckduckgo HTML search failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/infrastructure/web_search/duckduckgo.go","lineNumber":61,"sourceCode":"\tquery string,\n\tmaxResults int,\n\tincludeDate bool,\n) ([]*types.WebSearchResult, error) {\n\tif maxResults <= 0 {\n\t\tmaxResults = 5\n\t}\n\t// Try HTML scraping first (more reliable for general results)\n\thtmlResults, err := p.searchHTML(ctx, query, maxResults)\n\tif err == nil && len(htmlResults) > 0 {\n\t\treturn htmlResults, nil\n\t}\n\t// Fallback to Instant Answer API\n\tapiResults, apiErr := p.searchAPI(ctx, query, maxResults)\n\tif apiErr == nil && len(apiResults) > 0 {\n\t\treturn apiResults, nil\n\t}\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"duckduckgo HTML search failed: %w\", err)\n\t}\n\treturn nil, fmt.Errorf(\"duckduckgo API search failed: %w\", apiErr)\n}\n\n// searchHTML performs a web search using DuckDuckGo HTML endpoint\nfunc (p *DuckDuckGoProvider) searchHTML(\n\tctx context.Context,\n\tquery string,\n\tmaxResults int,\n) ([]*types.WebSearchResult, error) {\n\tbaseURL := \"https://html.duckduckgo.com/html/\"\n\tparams := url.Values{}\n\tparams.Set(\"q\", query)\n\tparams.Set(\"kl\", \"cn-zh\")\n\n\treqURL := baseURL + \"?\" + params.Encode()\n\treq, err := http.NewRequestWithContext(ctx, \"GET\", reqURL, nil)\n\tif err != nil {","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/infrastructure/web_search/duckduckgo.go#L43-L79","documentation":"DuckDuckGo's Search first tries the HTML endpoint; if that fails AND the Instant Answer API fallback also fails or returns no results, the original HTML error is wrapped as \"duckduckgo HTML search failed\". It indicates both retrieval paths failed, and the HTML error is reported as the root cause.","triggerScenarios":"searchHTML failed (non-200 status, network error, parse error) and then searchAPI returned an error or an empty result list, so the code falls back to reporting the original HTML failure.","commonSituations":"DuckDuckGo rate-limiting or bot-blocking the client (anomaly/captcha pages); no network egress; queries with no API answers; DKLM region param mismatch.","solutions":["Check the wrapped root cause (network error vs status code vs parse error) and address that first.","If blocked, add/refresh realistic headers or route through a proxy; DuckDuckGo aggressively blocks datacenter IPs.","Add delays/backoff between searches to avoid anomaly detection.","Consider a different search provider if DuckDuckGo scraping reliability is insufficient."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":"func isDDGHTMLFailure(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"duckduckgo HTML search failed\")\n}","tryCatchPattern":"results, err := ddg.Search(ctx, query, 10, false)\nif isDDGHTMLFailure(err) {\n    // both HTML and API paths failed; route to a different provider\n    results, err = bing.Search(ctx, query, 10, false)\n}","preventionTips":["Keep request headers and cadence realistic to avoid bot detection.","Use residential/proxy egress for scraping endpoints.","Chain multiple search providers so one blocked path doesn't fail the feature.","Throttle requests per host with a rate limiter."],"tags":["web-search","scraping","network","go"],"backgroundTag":"html-scrape-blocked","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}