{"record":{"id":"2f831ddea70412c6","repo":"charmbracelet/crush","slug":"failed-to-convert-html-to-markdown-w","errorCode":null,"errorMessage":"failed to convert HTML to markdown: %w","messagePattern":"failed to convert HTML to markdown: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/agent/tools/fetch_helpers.go","lineNumber":66,"sourceCode":"\tif err != nil {\n\t\treturn \"\", fmt.Errorf(\"failed to read response body: %w\", err)\n\t}\n\n\tcontent := string(body)\n\n\tif !utf8.ValidString(content) {\n\t\treturn \"\", errors.New(\"response content is not valid UTF-8\")\n\t}\n\n\tcontentType := resp.Header.Get(\"Content-Type\")\n\n\t// Convert HTML to markdown for better AI processing.\n\tif strings.Contains(contentType, \"text/html\") {\n\t\t// Remove noisy elements before conversion.\n\t\tcleanedHTML := removeNoisyElements(content)\n\t\tmarkdown, err := ConvertHTMLToMarkdown(cleanedHTML)\n\t\tif err != nil {\n\t\t\treturn \"\", fmt.Errorf(\"failed to convert HTML to markdown: %w\", err)\n\t\t}\n\t\tcontent = cleanupMarkdown(markdown)\n\t} else if strings.Contains(contentType, \"application/json\") || strings.Contains(contentType, \"text/json\") {\n\t\t// Format JSON for better readability.\n\t\tformatted, err := FormatJSON(content)\n\t\tif err == nil {\n\t\t\tcontent = formatted\n\t\t}\n\t\t// If formatting fails, keep original content.\n\t}\n\n\treturn content, nil\n}\n\n// removeNoisyElements removes script, style, nav, header, footer, and other\n// noisy elements from HTML to improve content extraction.\nfunc removeNoisyElements(htmlContent string) string {\n\tdoc, err := html.Parse(strings.NewReader(htmlContent))","sourceCodeStart":48,"sourceCodeEnd":84,"githubUrl":"https://github.com/charmbracelet/crush/blob/7944b8e52225d8805e31eacbf7ef24856b0dfb7a/internal/agent/tools/fetch_helpers.go#L48-L84","documentation":"The response body was identified as HTML (Content-Type contains text/html) but ConvertHTMLToMarkdown failed to parse/convert it. The raw HTML is cleaned of noisy elements first, then handed to the converter; malformed or exotic HTML that the converter cannot handle produces this wrapped error.","triggerScenarios":"Content-Type is text/html and ConvertHTMLToMarkdown(cleanedHTML) returns an error — typically severely malformed HTML, encoding issues, or a converter-internal failure.","commonSituations":"Scraping old or broken web pages with unclosed tags, pages served with wrong charsets, HTML fragments (not full documents), or pages that are actually error pages with odd markup.","solutions":["Check the wrapped converter error for the specific parse failure","Verify the page renders valid HTML (validate with an HTML linter or W3C validator)","If conversion repeatedly fails, fall back to returning the cleaned raw HTML or plain text","Ensure removeNoisyElements is not corrupting otherwise-valid HTML"],"exampleFix":"// before\nmarkdown, err := ConvertHTMLToMarkdown(cleanedHTML)\nif err != nil {\n    return \"\", fmt.Errorf(\"failed to convert HTML to markdown: %w\", err)\n}\n// after\nmarkdown, err := ConvertHTMLToMarkdown(cleanedHTML)\nif err != nil {\n    // Fall back to cleaned raw HTML instead of failing the whole fetch.\n    return cleanupMarkdown(cleanedHTML), nil\n}","handlingStrategy":"fallback","validationCode":"ct := resp.Header.Get(\"Content-Type\")\nif strings.Contains(ct, \"text/html\") {\n    // proceed with conversion, prepared to fall back\n}","typeGuard":"null","tryCatchPattern":"markdown, err := ConvertHTMLToMarkdown(cleanedHTML)\nif err != nil {\n    // fall back to cleaned raw HTML or plain text instead of failing\n    return cleanupMarkdown(cleanedHTML), nil\n}","preventionTips":["Sanitize/clean HTML before conversion (already done via removeNoisyElements)","Keep the HTML-to-markdown converter updated for malformed input tolerance","Fall back to raw text when conversion fails rather than erroring the whole fetch","Check Content-Type before choosing the conversion path"],"tags":["html","markdown","conversion"],"backgroundTag":"html-to-markdown-conversion-failed","analyzedSha":"7944b8e52225d8805e31eacbf7ef24856b0dfb7a","analyzedAt":"2026-08-29T12:48:59.079Z","schemaVersion":2},"datasetVersion":"2026-08-29T17:17:51.833Z"}