Tencent/WeKnora · error

failed to parse HTML: %w

Error message

failed to parse HTML: %w

What it means

goquery.NewDocumentFromReader failed while parsing the DuckDuckGo HTML response body in searchHTML, wrapped as "failed to parse HTML". The response arrived with a 200/202 status but could not be parsed as HTML (usually because it isn't the expected page).

Source

Thrown at internal/infrastructure/web_search/duckduckgo.go:105

	curlCommand := fmt.Sprintf(
		"curl -X GET '%s' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'",
		req.URL.String(),
	)
	logger.Infof(ctx, "Curl of request: %s", secutils.SanitizeForLog(curlCommand))

	resp, err := p.client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("failed to perform request: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted {
		return nil, fmt.Errorf("duckduckgo HTML returned status %d", resp.StatusCode)
	}

	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("failed to parse HTML: %w", err)
	}

	results := make([]*types.WebSearchResult, 0, maxResults)
	doc.Find(".web-result").Each(func(i int, s *goquery.Selection) {
		if len(results) >= maxResults {
			return
		}
		titleNode := s.Find(".result__a")
		title := strings.TrimSpace(titleNode.Text())
		var link string
		if href, exists := titleNode.Attr("href"); exists {
			link = cleanDDGURL(href)
		}
		snippet := strings.TrimSpace(s.Find(".result__snippet").Text())
		if title != "" && link != "" {
			results = append(results, &types.WebSearchResult{
				Title:   title,
				URL:     link,

View on GitHub (pinned to 988cbb0330)

Solutions

1. Capture the raw body (e.g. read it fully and log a prefix) to see what was actually returned before parsing.
2. Check for bot-challenge/anomaly pages even on 200 responses and treat them as block signals.
3. Ensure the response body isn't read or closed twice before goquery sees it.
4. Verify Content-Encoding handling (avoid decompressing twice).

Example fix

// before
doc, err := goquery.NewDocumentFromReader(resp.Body)
// after
bodyBytes, readErr := io.ReadAll(resp.Body)
if readErr != nil {
    return nil, fmt.Errorf("failed to read body: %w", readErr)
}
if len(bytes.TrimSpace(bodyBytes)) == 0 {
    return nil, errors.New("empty response body from duckduckgo")
}
doc, err := goquery.NewDocumentFromReader(bytes.NewReader(bodyBytes))

func isParseFailure(err error) bool {
    return err != nil && strings.Contains(err.Error(), "failed to parse HTML")
}

results, err := ddg.Search(ctx, query, 10, false)
if isParseFailure(err) {
    // unexpected body (challenge page?) — fall back to API or another provider
    return apiProvider.Search(ctx, query, 10, false)
}

When it happens

Trigger: Body is empty, truncated, gzip/cbor encoded oddly, or is a challenge page that still returns 200 but with malformed HTML, breaking goquery's parse step.

Common situations: Response body already consumed or closed; proxy injecting garbage; DuckDuckGo serving a JS-only or anomaly page with a 200 status; nil or errored reader state.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to unmarshal response: %w
• duckduckgo HTML search failed: %w
• duckduckgo HTML returned status %d
• readability parse: %w
• query is empty

AI-assisted analysis of Tencent/WeKnora@988cbb0330 (2026-09-02). Data as JSON: /api/errors/892006609c518a18. Report an issue: GitHub.