fish2018/pansou · error

[ ] 网页搜索 HTML 解析失败

Error message

[%s] 网页搜索 HTML 解析失败: %w

What it means

The feikuai plugin falls back to scraping a web search page when its primary API call fails. goquery could not parse the HTML response body returned by the fallback search endpoint. This indicates the response was not valid/complete HTML (blocked page, truncated body, or non-HTML content type).

Solutions

1. Retry the request; transient truncation often succeeds on a second attempt.
2. Check the fallback search endpoint with curl to see what content is actually returned (captcha page, JSON, empty body).
3. Set a realistic User-Agent/headers so the site does not serve an anti-bot page.
4. Improve API reliability (credentials, rate limits) so the HTML fallback is not needed.

Example fix

// before
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
    return nil, fmt.Errorf("[%s] 网页搜索 HTML 解析失败: %w", p.Name(), err)
}
// after
doc, err := goquery.NewDocumentFromReader(io.LimitReader(resp.Body, 5<<20))
if err != nil {
    log.Printf("[%s] fallback HTML parse failed (status %d, content-type %s)", p.Name(), resp.StatusCode, resp.Header.Get("Content-Type"))
    return nil, fmt.Errorf("[%s] 网页搜索 HTML 解析失败: %w", p.Name(), err)
}

// before calling search, check the site responds with HTML
resp, err := http.Get(fallbackURL)
if err == nil {
    ct := resp.Header.Get("Content-Type")
    if !strings.Contains(ct, "text/html") {
        // skip fallback scrape
    }
}

func isHTMLResponse(resp *http.Response) bool {
    return resp != nil && strings.Contains(resp.Header.Get("Content-Type"), "text/html")
}

results, err := plugin.Search(keyword)
if err != nil {
    var parseErr *goqueryError // or use errors.Is/As on wrapped causes
    log.Printf("search failed (fallback HTML unparseable): %v", err)
    results = []model.SearchResult{} // degrade gracefully
}

When it happens

Trigger: searchWeb: the API attempt failed (apiErr set), the fallback web-search HTTP response returned status 200, but goquery.NewDocumentFromReader failed to parse resp.Body's HTML.

Common situations: Site returns a captcha/challenge page with malformed HTML, CDN truncates the response, server returns JSON or binary instead of HTML on the fallback URL, network proxy corrupts the body.

Related errors

• [ ] 解析搜索结果失败
• [ ] 解析搜索页面失败
• [ ] 解析HTML失败
• [ ] 解析搜索结果HTML失败
• [ ] 解析搜索结果HTML失败

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/ef6c4bf2a4aa76be. Report an issue: GitHub.

Appendix: source

Thrown at plugin/feikuai/feikuai.go:193

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, searchURL, nil)
	if err != nil {
		return nil, fmt.Errorf("[%s] 创建网页搜索请求失败: %w", p.Name(), err)
	}
	req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/136.0.0.0 Safari/537.36")
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
	req.Header.Set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8")
	req.Header.Set("Referer", "https://feikuai.in/")
	resp, err := client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("[%s] API 失败且网页搜索请求失败: %v (API: %v)", p.Name(), err, apiErr)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("[%s] API 失败且网页搜索返回状态码 %d (API: %v)", p.Name(), resp.StatusCode, apiErr)
	}
	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("[%s] 网页搜索 HTML 解析失败: %w", p.Name(), err)
	}

	results := make([]model.SearchResult, 0, 64)
	seen := make(map[string]struct{})
	add := func(linkURL, title, content string, datetime time.Time) {
		linkURL = strings.TrimSpace(linkURL)
		if linkURL == "" {
			return
		}
		linkType := util.GetLinkType(linkURL)
		if linkType == "" || linkType == "others" {
			return
		}
		if _, ok := seen[linkURL]; ok {
			return
		}
		seen[linkURL] = struct{}{}
		if title == "" {

View on GitHub (pinned to beaa561337)