charmbracelet/crush · error

failed to convert HTML to markdown: %w

Error message

failed to convert HTML to markdown: %w

What it means

The response body was identified as HTML (Content-Type contains text/html) but ConvertHTMLToMarkdown failed to parse/convert it. The raw HTML is cleaned of noisy elements first, then handed to the converter; malformed or exotic HTML that the converter cannot handle produces this wrapped error.

Source

Thrown at internal/agent/tools/fetch_helpers.go:66

	if err != nil {
		return "", fmt.Errorf("failed to read response body: %w", err)
	}

	content := string(body)

	if !utf8.ValidString(content) {
		return "", errors.New("response content is not valid UTF-8")
	}

	contentType := resp.Header.Get("Content-Type")

	// Convert HTML to markdown for better AI processing.
	if strings.Contains(contentType, "text/html") {
		// Remove noisy elements before conversion.
		cleanedHTML := removeNoisyElements(content)
		markdown, err := ConvertHTMLToMarkdown(cleanedHTML)
		if err != nil {
			return "", fmt.Errorf("failed to convert HTML to markdown: %w", err)
		}
		content = cleanupMarkdown(markdown)
	} else if strings.Contains(contentType, "application/json") || strings.Contains(contentType, "text/json") {
		// Format JSON for better readability.
		formatted, err := FormatJSON(content)
		if err == nil {
			content = formatted
		}
		// If formatting fails, keep original content.
	}

	return content, nil
}

// removeNoisyElements removes script, style, nav, header, footer, and other
// noisy elements from HTML to improve content extraction.
func removeNoisyElements(htmlContent string) string {
	doc, err := html.Parse(strings.NewReader(htmlContent))

View on GitHub (pinned to 7944b8e522)

Solutions

1. Check the wrapped converter error for the specific parse failure
2. Verify the page renders valid HTML (validate with an HTML linter or W3C validator)
3. If conversion repeatedly fails, fall back to returning the cleaned raw HTML or plain text
4. Ensure removeNoisyElements is not corrupting otherwise-valid HTML

Example fix

// before
markdown, err := ConvertHTMLToMarkdown(cleanedHTML)
if err != nil {
    return "", fmt.Errorf("failed to convert HTML to markdown: %w", err)
}
// after
markdown, err := ConvertHTMLToMarkdown(cleanedHTML)
if err != nil {
    // Fall back to cleaned raw HTML instead of failing the whole fetch.
    return cleanupMarkdown(cleanedHTML), nil
}

ct := resp.Header.Get("Content-Type")
if strings.Contains(ct, "text/html") {
    // proceed with conversion, prepared to fall back
}

null

markdown, err := ConvertHTMLToMarkdown(cleanedHTML)
if err != nil {
    // fall back to cleaned raw HTML or plain text instead of failing
    return cleanupMarkdown(cleanedHTML), nil
}

When it happens

Trigger: Content-Type is text/html and ConvertHTMLToMarkdown(cleanedHTML) returns an error — typically severely malformed HTML, encoding issues, or a converter-internal failure.

Common situations: Scraping old or broken web pages with unclosed tags, pages served with wrong charsets, HTML fragments (not full documents), or pages that are actually error pages with odd markup.

Related errors

• failed to parse HTML: %w
• failed to generate HTML: %w

AI-assisted analysis of charmbracelet/crush@7944b8e522 (2026-08-29). Data as JSON: /api/errors/2f831ddea70412c6. Report an issue: GitHub.