{"record":{"id":"1f9291a13d4a53ae","repo":"vxcontrol/pentagi","slug":"failed-to-parse-search-response-w","errorCode":null,"errorMessage":"failed to parse search response: %w","messagePattern":"failed to parse search response: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"backend/pkg/tools/searchers/duckduckgo.go","lineNumber":193,"sourceCode":"\t\t\t\treturn \"\", fmt.Errorf(\"unexpected status code: %d\", resp.StatusCode)\n\t\t\t}\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn \"\", ctx.Err()\n\t\t\tcase <-time.After(time.Second):\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tbody, err := io.ReadAll(resp.Body)\n\t\tresp.Body.Close()\n\t\tif err != nil {\n\t\t\treturn \"\", fmt.Errorf(\"failed to read response body: %w\", err)\n\t\t}\n\n\t\tresponse, err = d.parseHTMLResponse(body)\n\t\tif err != nil {\n\t\t\treturn \"\", fmt.Errorf(\"failed to parse search response: %w\", err)\n\t\t}\n\n\t\tbreak\n\t}\n\n\tif response == nil || len(response.Results) == 0 {\n\t\treturn \"No results found\", nil\n\t}\n\n\t// Limit results to requested number\n\tif len(response.Results) > maxResults {\n\t\tresponse.Results = response.Results[:maxResults]\n\t}\n\n\t// Format results in readable text format\n\treturn d.formatSearchResults(response.Results), nil\n}\n","sourceCodeStart":175,"sourceCodeEnd":211,"githubUrl":"https://github.com/vxcontrol/pentagi/blob/ea665308baaff015b226f308438a68d929d0f29b/backend/pkg/tools/searchers/duckduckgo.go#L175-L211","documentation":"parseHTMLResponse failed: both the structured x/net/html extraction and the regex fallback returned an error for the DuckDuckGo response body. Note the structured parser only errors on html.Parse failure (rare for real HTML); in practice this usually means the regex fallback errored because the page doesn't look like a results page at all.","triggerScenarios":"DuckDuckGo returned a 200 page that is not a search-results page — a bot-challenge/anomaly page, a CAPTCHA interstitial, or a completely redesigned layout — so neither parser can extract results and the fallback regex parser errors.","commonSituations":"DuckDuckGo serving challenge pages to datacenter IPs; scraping the HTML endpoint after DuckDuckGo changes markup or serves a JS-rendered page; proxy returning an interstitial page with status 200.","solutions":["Dump and inspect the raw body (log a snippet on parse failure) to see what page was actually returned.","If it's a bot-challenge page, rotate egress IP / reduce request rate or switch engines.","If DuckDuckGo changed its HTML, update the CSS-class selectors in findResultNodes and the patterns in parseHTMLRegex.","Ensure the fallback regex parser returns nil (not error) for empty pages if zero results should be a 'No results found' outcome rather than an error."],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// detect challenge pages before parsing\nif strings.Contains(string(body), \"anomaly\") || strings.Contains(string(body), \"challenge\") {\n    return nil, errors.New(\"duckduckgo returned a bot-challenge page\")\n}","typeGuard":null,"tryCatchPattern":"response, err := d.parseHTMLResponse(body)\nif err != nil {\n    // fall back to next engine rather than failing the whole web_search\n    return \"\", Fatal(fmt.Errorf(\"failed to parse search response: %w\", err))\n}","preventionTips":["Log a snippet of unparseable bodies to detect layout changes and challenge pages early","Keep the structured and regex parsers both resilient to markup drift","Return empty results (not error) for valid non-results pages so 'No results found' is handled as success","Add a pinned fixture test that parses a saved DuckDuckGo results page in CI"],"tags":["html","parsing","scraping","duckduckgo"],"backgroundTag":"html-parse-failed","analyzedSha":"ea665308baaff015b226f308438a68d929d0f29b","analyzedAt":"2026-09-01T14:16:31.421Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}