vxcontrol/pentagi · warning

failed to parse HTML: %w

Error message

failed to parse HTML: %w

What it means

parseHTMLStructured calls golang.org/x/net/html's html.Parse on the response body. html.Parse is extremely lenient — it can error only on unrecoverable internal conditions (e.g. very deeply nested markup overflowing the parser's limits), since it otherwise repairs malformed HTML. This error is usually swallowed by the caller: parseHTMLResponse falls back to regex parsing when the structured pass yields no results.

Source

Thrown at backend/pkg/tools/searchers/duckduckgo.go:261

	}

	// Fallback to regex-based parsing
	results, err = d.parseHTMLRegex(body)
	if err != nil {
		return nil, err
	}

	return &searchResponse{
		Results:   results,
		NoResults: len(results) == 0,
	}, nil
}

// parseHTMLStructured uses golang.org/x/net/html for structured HTML parsing
func (d *duckduckgo) parseHTMLStructured(body []byte) ([]searchResult, error) {
	doc, err := html.Parse(strings.NewReader(string(body)))
	if err != nil {
		return nil, fmt.Errorf("failed to parse HTML: %w", err)
	}

	results := make([]searchResult, 0)
	d.findResultNodes(doc, &results)

	return results, nil
}

// findResultNodes recursively finds and extracts search result nodes
func (d *duckduckgo) findResultNodes(n *html.Node, results *[]searchResult) {
	// Look for div with class "result results_links"
	if n.Type == html.ElementNode && n.Data == "div" {
		if d.hasClass(n, "result") && d.hasClass(n, "results_links") {
			result := d.extractResultFromNode(n)
			if result.Title != "" && result.URL != "" {
				*results = append(*results, result)
			}
		}

View on GitHub (pinned to ea665308ba)

Solutions

1. Rely on the existing fallback: parseHTMLResponse already retries with parseHTMLRegex, so confirm the regex path succeeded.
2. Inspect the body if this recurs — corruption mid-transfer usually indicates network/proxy problems rather than a parser bug.
3. Update golang.org/x/net if a known html.Parse bug matches the wrapped error.
4. Log the wrapped error text; it pinpoints whether it's a depth limit or tokenizer failure.

// reject obviously non-HTML bodies before parsing
trimmed := bytes.TrimSpace(body)
if len(trimmed) > 0 && trimmed[0] != '<' {
    return nil, errors.New("response body is not HTML")
}

results, err := d.parseHTMLStructured(body)
if err != nil {
    // structured parse failed; regex fallback in parseHTMLResponse handles it
    log.WithError(err).Debug("structured parse failed, falling back to regex")
    return d.parseHTMLRegex(body)
}

When it happens

Trigger: html.Parse returns a non-nil error for the DuckDuckGo response — practically only with pathologically deep nesting in the body (html.ParseError with huge depth), truncated/corrupt responses, or a non-HTML body large/degenerate enough to break the tokenizer.

Common situations: Extremely rare in production; seen with truncated/corrupted response bodies, binary garbage served with 200 by a misbehaving proxy, or x/net/html version regressions on pathological pages.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• failed to parse search response: %w
• bearer scheme must be used
• token can't be empty
• token validation disabled with default salt
• token is invalid

AI-assisted analysis of vxcontrol/pentagi@ea665308ba (2026-09-01). Data as JSON: /api/errors/154d7676b19271cf. Report an issue: GitHub.