{"record":{"id":"892006609c518a18","repo":"Tencent/WeKnora","slug":"failed-to-parse-html-w","errorCode":null,"errorMessage":"failed to parse HTML: %w","messagePattern":"failed to parse HTML: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/infrastructure/web_search/duckduckgo.go","lineNumber":105,"sourceCode":"\tcurlCommand := fmt.Sprintf(\n\t\t\"curl -X GET '%s' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'\",\n\t\treq.URL.String(),\n\t)\n\tlogger.Infof(ctx, \"Curl of request: %s\", secutils.SanitizeForLog(curlCommand))\n\n\tresp, err := p.client.Do(req)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to perform request: %w\", err)\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted {\n\t\treturn nil, fmt.Errorf(\"duckduckgo HTML returned status %d\", resp.StatusCode)\n\t}\n\n\tdoc, err := goquery.NewDocumentFromReader(resp.Body)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to parse HTML: %w\", err)\n\t}\n\n\tresults := make([]*types.WebSearchResult, 0, maxResults)\n\tdoc.Find(\".web-result\").Each(func(i int, s *goquery.Selection) {\n\t\tif len(results) >= maxResults {\n\t\t\treturn\n\t\t}\n\t\ttitleNode := s.Find(\".result__a\")\n\t\ttitle := strings.TrimSpace(titleNode.Text())\n\t\tvar link string\n\t\tif href, exists := titleNode.Attr(\"href\"); exists {\n\t\t\tlink = cleanDDGURL(href)\n\t\t}\n\t\tsnippet := strings.TrimSpace(s.Find(\".result__snippet\").Text())\n\t\tif title != \"\" && link != \"\" {\n\t\t\tresults = append(results, &types.WebSearchResult{\n\t\t\t\tTitle:   title,\n\t\t\t\tURL:     link,","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/Tencent/WeKnora/blob/988cbb03305e055d8ebb7d46d9ac6cc0803cd074/internal/infrastructure/web_search/duckduckgo.go#L87-L123","documentation":"goquery.NewDocumentFromReader failed while parsing the DuckDuckGo HTML response body in searchHTML, wrapped as \"failed to parse HTML\". The response arrived with a 200/202 status but could not be parsed as HTML (usually because it isn't the expected page).","triggerScenarios":"Body is empty, truncated, gzip/cbor encoded oddly, or is a challenge page that still returns 200 but with malformed HTML, breaking goquery's parse step.","commonSituations":"Response body already consumed or closed; proxy injecting garbage; DuckDuckGo serving a JS-only or anomaly page with a 200 status; nil or errored reader state.","solutions":["Capture the raw body (e.g. read it fully and log a prefix) to see what was actually returned before parsing.","Check for bot-challenge/anomaly pages even on 200 responses and treat them as block signals.","Ensure the response body isn't read or closed twice before goquery sees it.","Verify Content-Encoding handling (avoid decompressing twice)."],"exampleFix":"// before\ndoc, err := goquery.NewDocumentFromReader(resp.Body)\n// after\nbodyBytes, readErr := io.ReadAll(resp.Body)\nif readErr != nil {\n    return nil, fmt.Errorf(\"failed to read body: %w\", readErr)\n}\nif len(bytes.TrimSpace(bodyBytes)) == 0 {\n    return nil, errors.New(\"empty response body from duckduckgo\")\n}\ndoc, err := goquery.NewDocumentFromReader(bytes.NewReader(bodyBytes))","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"func isParseFailure(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"failed to parse HTML\")\n}","tryCatchPattern":"results, err := ddg.Search(ctx, query, 10, false)\nif isParseFailure(err) {\n    // unexpected body (challenge page?) — fall back to API or another provider\n    return apiProvider.Search(ctx, query, 10, false)\n}","preventionTips":["Log a truncated copy of raw bodies in debug mode to diagnose unexpected pages.","Treat 200-with-challenge-page as a block signal and back off.","Guard against double reads/closes of resp.Body in custom clients.","Add regression tests parsing saved copies of real DuckDuckGo result pages."],"tags":["html","parsing","scraping","web-search","go"],"backgroundTag":"html-parse-failed","analyzedSha":"988cbb03305e055d8ebb7d46d9ac6cc0803cd074","analyzedAt":"2026-09-02T14:41:08.344Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}