What causes "failed to parse search response: %w" in vxcontrol/pentagi?

Dump and inspect the raw body (log a snippet on parse failure) to see what page was actually returned.

failed to parse search response: %w

Error message

failed to parse search response: %w

What it means

parseHTMLResponse failed: both the structured x/net/html extraction and the regex fallback returned an error for the DuckDuckGo response body. Note the structured parser only errors on html.Parse failure (rare for real HTML); in practice this usually means the regex fallback errored because the page doesn't look like a results page at all.

Source

Thrown at backend/pkg/tools/searchers/duckduckgo.go:193

				return "", fmt.Errorf("unexpected status code: %d", resp.StatusCode)
			}
			select {
			case <-ctx.Done():
				return "", ctx.Err()
			case <-time.After(time.Second):
			}
			continue
		}

		body, err := io.ReadAll(resp.Body)
		resp.Body.Close()
		if err != nil {
			return "", fmt.Errorf("failed to read response body: %w", err)
		}

		response, err = d.parseHTMLResponse(body)
		if err != nil {
			return "", fmt.Errorf("failed to parse search response: %w", err)
		}

		break
	}

	if response == nil || len(response.Results) == 0 {
		return "No results found", nil
	}

	// Limit results to requested number
	if len(response.Results) > maxResults {
		response.Results = response.Results[:maxResults]
	}

	// Format results in readable text format
	return d.formatSearchResults(response.Results), nil
}

View on GitHub (pinned to ea665308ba)

Solutions

Dump and inspect the raw body (log a snippet on parse failure) to see what page was actually returned.
If it's a bot-challenge page, rotate egress IP / reduce request rate or switch engines.
If DuckDuckGo changed its HTML, update the CSS-class selectors in findResultNodes and the patterns in parseHTMLRegex.
Ensure the fallback regex parser returns nil (not error) for empty pages if zero results should be a 'No results found' outcome rather than an error.

Defensive patterns

Strategy: fallback

Validate before calling

// detect challenge pages before parsing
if strings.Contains(string(body), "anomaly") || strings.Contains(string(body), "challenge") {
    return nil, errors.New("duckduckgo returned a bot-challenge page")
}

Try / catch

response, err := d.parseHTMLResponse(body)
if err != nil {
    // fall back to next engine rather than failing the whole web_search
    return "", Fatal(fmt.Errorf("failed to parse search response: %w", err))
}

Prevention

Log a snippet of unparseable bodies to detect layout changes and challenge pages early
Keep the structured and regex parsers both resilient to markup drift
Return empty results (not error) for valid non-results pages so 'No results found' is handled as success
Add a pinned fixture test that parses a saved DuckDuckGo results page in CI

When it happens

Trigger: DuckDuckGo returned a 200 page that is not a search-results page — a bot-challenge/anomaly page, a CAPTCHA interstitial, or a completely redesigned layout — so neither parser can extract results and the fallback regex parser errors.

Common situations: DuckDuckGo serving challenge pages to datacenter IPs; scraping the HTML endpoint after DuckDuckGo changes markup or serves a JS-rendered page; proxy returning an interstitial page with status 200.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/1f9291a13d4a53ae. Report an issue: GitHub.