What causes "failed to parse HTML: %w" in charmbracelet/crush?

Log the first ~200 bytes of htmlContent when this error occurs to see what was actually received.

failed to parse HTML: %w

Error message

failed to parse HTML: %w

What it means

Wraps a golang.org/x/net/html Parse error in parseLiteSearchResults (internal/agent/tools/search.go:133-136). The DuckDuckGo Lite HTML could not be tokenized into a DOM. html.Parse is extremely lenient, so this only fires on truly unparseable input — usually empty or binary/garbage data rather than merely malformed markup.

Source

Thrown at internal/agent/tools/search.go:135

	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
	req.Header.Set("Accept-Language", acceptLanguages[rand.IntN(len(acceptLanguages))])
	req.Header.Set("Accept-Encoding", "identity")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Upgrade-Insecure-Requests", "1")
	req.Header.Set("Sec-Fetch-Dest", "document")
	req.Header.Set("Sec-Fetch-Mode", "navigate")
	req.Header.Set("Sec-Fetch-Site", "none")
	req.Header.Set("Sec-Fetch-User", "?1")
	req.Header.Set("Cache-Control", "max-age=0")
	if rand.IntN(2) == 0 {
		req.Header.Set("DNT", "1")
	}
}

func parseLiteSearchResults(htmlContent string, maxResults int) ([]SearchResult, error) {
	doc, err := html.Parse(strings.NewReader(htmlContent))
	if err != nil {
		return nil, fmt.Errorf("failed to parse HTML: %w", err)
	}

	var results []SearchResult
	var currentResult *SearchResult

	var traverse func(*html.Node)
	traverse = func(n *html.Node) {
		if n.Type == html.ElementNode {
			if n.Data == "a" && hasClass(n, "result-link") {
				if currentResult != nil && currentResult.Link != "" {
					currentResult.Position = len(results) + 1
					results = append(results, *currentResult)
					if len(results) >= maxResults {
						return
					}
				}
				currentResult = &SearchResult{Title: getTextContent(n)}
				for _, attr := range n.Attr {

View on GitHub (pinned to 7944b8e522)

Solutions

Log the first ~200 bytes of htmlContent when this error occurs to see what was actually received.
Verify the response is decompressed: the client sets Accept-Encoding: identity, so a proxy forcing gzip will produce garbage — disable such proxy rewriting.
Check whether a captive portal or security appliance is intercepting lite.duckduckgo.com.
If feeding test data, ensure the fixture contains valid (even loose) HTML, not an empty string or JSON.

Example fix

null

Defensive patterns

Strategy: validation

Validate before calling

if strings.TrimSpace(htmlContent) == "" {
    return nil, fmt.Errorf("empty HTML from DuckDuckGo, likely blocked or truncated")
}
if !utf8.ValidString(htmlContent) {
    return nil, fmt.Errorf("non-UTF-8 body, likely compressed or binary content")
}

Try / catch

results, err := parseLiteSearchResults(content, maxResults)
if err != nil {
    log.Printf("HTML parse failed, body head: %.200q", content)
    return nil, err
}

Prevention

Verify Accept-Encoding handling keeps identity so bodies arrive uncompressed.
Log response body prefixes on parse failures to detect proxies and captive portals.
Validate test fixtures contain real (even loose) HTML before feeding the parser.
Check for interception appliances when the error repeats on one network.

When it happens

Trigger: html.Parse receives a string that cannot be parsed: an empty string, compressed bytes (e.g. gzip body decoded wrongly because Accept-Encoding handling changed), or non-HTML binary content substituted for the results page by a proxy or captive portal.

Common situations: A captive portal or proxy replacing the response with compressed or binary data; a test httptest server (see ddgLiteEndpoint override) returning empty or non-HTML bodies; upstream changes where the body arrives gzip-encoded and is not decompressed before parsing.

Understand the failure class

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of charmbracelet/crush@7944b8e522 (2026-08-29). Data as JSON: /api/errors/ea4f08c191e2705b. Report an issue: GitHub.