{"record":{"id":"ea4f08c191e2705b","repo":"charmbracelet/crush","slug":"failed-to-parse-html-w","errorCode":null,"errorMessage":"failed to parse HTML: %w","messagePattern":"failed to parse HTML: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/agent/tools/search.go","lineNumber":135,"sourceCode":"\treq.Header.Set(\"Accept\", \"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\")\n\treq.Header.Set(\"Accept-Language\", acceptLanguages[rand.IntN(len(acceptLanguages))])\n\treq.Header.Set(\"Accept-Encoding\", \"identity\")\n\treq.Header.Set(\"Connection\", \"keep-alive\")\n\treq.Header.Set(\"Upgrade-Insecure-Requests\", \"1\")\n\treq.Header.Set(\"Sec-Fetch-Dest\", \"document\")\n\treq.Header.Set(\"Sec-Fetch-Mode\", \"navigate\")\n\treq.Header.Set(\"Sec-Fetch-Site\", \"none\")\n\treq.Header.Set(\"Sec-Fetch-User\", \"?1\")\n\treq.Header.Set(\"Cache-Control\", \"max-age=0\")\n\tif rand.IntN(2) == 0 {\n\t\treq.Header.Set(\"DNT\", \"1\")\n\t}\n}\n\nfunc parseLiteSearchResults(htmlContent string, maxResults int) ([]SearchResult, error) {\n\tdoc, err := html.Parse(strings.NewReader(htmlContent))\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to parse HTML: %w\", err)\n\t}\n\n\tvar results []SearchResult\n\tvar currentResult *SearchResult\n\n\tvar traverse func(*html.Node)\n\ttraverse = func(n *html.Node) {\n\t\tif n.Type == html.ElementNode {\n\t\t\tif n.Data == \"a\" && hasClass(n, \"result-link\") {\n\t\t\t\tif currentResult != nil && currentResult.Link != \"\" {\n\t\t\t\t\tcurrentResult.Position = len(results) + 1\n\t\t\t\t\tresults = append(results, *currentResult)\n\t\t\t\t\tif len(results) >= maxResults {\n\t\t\t\t\t\treturn\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tcurrentResult = &SearchResult{Title: getTextContent(n)}\n\t\t\t\tfor _, attr := range n.Attr {","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/charmbracelet/crush/blob/7944b8e52225d8805e31eacbf7ef24856b0dfb7a/internal/agent/tools/search.go#L117-L153","documentation":"Wraps a golang.org/x/net/html Parse error in parseLiteSearchResults (internal/agent/tools/search.go:133-136). The DuckDuckGo Lite HTML could not be tokenized into a DOM. html.Parse is extremely lenient, so this only fires on truly unparseable input — usually empty or binary/garbage data rather than merely malformed markup.","triggerScenarios":"html.Parse receives a string that cannot be parsed: an empty string, compressed bytes (e.g. gzip body decoded wrongly because Accept-Encoding handling changed), or non-HTML binary content substituted for the results page by a proxy or captive portal.","commonSituations":"A captive portal or proxy replacing the response with compressed or binary data; a test httptest server (see ddgLiteEndpoint override) returning empty or non-HTML bodies; upstream changes where the body arrives gzip-encoded and is not decompressed before parsing.","solutions":["Log the first ~200 bytes of htmlContent when this error occurs to see what was actually received.","Verify the response is decompressed: the client sets Accept-Encoding: identity, so a proxy forcing gzip will produce garbage — disable such proxy rewriting.","Check whether a captive portal or security appliance is intercepting lite.duckduckgo.com.","If feeding test data, ensure the fixture contains valid (even loose) HTML, not an empty string or JSON."],"exampleFix":"null","handlingStrategy":"validation","validationCode":"if strings.TrimSpace(htmlContent) == \"\" {\n    return nil, fmt.Errorf(\"empty HTML from DuckDuckGo, likely blocked or truncated\")\n}\nif !utf8.ValidString(htmlContent) {\n    return nil, fmt.Errorf(\"non-UTF-8 body, likely compressed or binary content\")\n}","typeGuard":null,"tryCatchPattern":"results, err := parseLiteSearchResults(content, maxResults)\nif err != nil {\n    log.Printf(\"HTML parse failed, body head: %.200q\", content)\n    return nil, err\n}","preventionTips":["Verify Accept-Encoding handling keeps identity so bodies arrive uncompressed.","Log response body prefixes on parse failures to detect proxies and captive portals.","Validate test fixtures contain real (even loose) HTML before feeding the parser.","Check for interception appliances when the error repeats on one network."],"tags":["html","parsing","go"],"backgroundTag":"html-parse-failed","analyzedSha":"7944b8e52225d8805e31eacbf7ef24856b0dfb7a","analyzedAt":"2026-08-29T12:48:59.079Z","schemaVersion":2},"datasetVersion":"2026-08-29T17:17:51.833Z"}