fish2018/pansou · error

[ ] 解析搜索页面失败

Error message

[%s] 解析搜索页面失败: %w

What it means

goquery.NewDocumentFromReader failed to parse the HTTP response body as HTML after a successful 200 response from the duoduo search page. goquery wraps golang.org/x/net/html, which returns an error on badly malformed markup or an empty/invalid body. The plugin wraps it with the plugin name for context.

Solutions

1. Log the first few hundred bytes of resp.Body (after io.ReadAll) to see what content actually arrived.
2. Ensure the http.Client (not a manual io.Copy) handles the response so automatic gzip decompression applies; do not set Accept-Encoding manually.
3. Check resp.Header Content-Type is text/html before parsing; fall back or error clearly otherwise.
4. Retry the request if the body is empty — transient truncation is common with flaky upstreams.

Example fix

// before
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
    return nil, fmt.Errorf("[%s] 解析搜索页面失败: %w", p.Name(), err)
}
// after
body, _ := io.ReadAll(resp.Body)
if len(bytes.TrimSpace(body)) == 0 {
    return nil, fmt.Errorf("[%s] 搜索页面响应为空", p.Name())
}
doc, err := goquery.NewDocumentFromReader(bytes.NewReader(body))
if err != nil {
    return nil, fmt.Errorf("[%s] 解析搜索页面失败: %w", p.Name(), err)
}

body, err := io.ReadAll(resp.Body)
if err != nil { return err }
if len(bytes.TrimSpace(body)) == 0 { return fmt.Errorf("empty body") }
if ct := resp.Header.Get("Content-Type"); !strings.Contains(ct, "text/html") {
    return fmt.Errorf("unexpected content-type: %s", ct)
}
doc, err := goquery.NewDocumentFromReader(bytes.NewReader(body))

func looksLikeHTML(b []byte) bool {
    s := bytes.TrimSpace(b)
    return len(s) > 0 && (bytes.Contains(bytes.ToLower(s[:min(200, len(s))], []byte("<html")) || bytes.Contains(bytes.ToLower(s[:min(200, len(s))]), []byte("<!doctype"))
}

doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
    // log a body preview, then return a clear parse error
    return nil, fmt.Errorf("parse search page: %w", err)
}

When it happens

Trigger: The response body is not valid HTML — e.g. an empty body, a JSON error payload returned with status 200, gzip/brotli-compressed bytes that were not decompressed because Content-Encoding handling was bypassed, or a truncated body from a connection drop mid-read.

Common situations: Site serves compressed responses but a custom transport strips automatic decompression; anti-bot systems return a 200 with a JS-challenge or empty page; server returns non-HTML content (JSON/XML) due to changed API; network interruption truncates the response.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

• [ ] 解析首页失败
• [ ] HTML解析失败
• [ ] 解析搜索页面失败
• [ ] HTML解析失败
• [ ] 解析详情页失败

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/df26f8f645110a86. Report an issue: GitHub.

Appendix: source

Thrown at plugin/duoduo/duoduo.go:190

	req.Header.Set("Upgrade-Insecure-Requests", "1")
	req.Header.Set("Cache-Control", "max-age=0")
	req.Header.Set("Referer", "https://tv.yydsys.top/")
	
	// 5. 发送请求（带重试机制）
	resp, err := p.doRequestWithRetry(req, client)
	if err != nil {
		return nil, fmt.Errorf("[%s] 搜索请求失败: %w", p.Name(), err)
	}
	defer resp.Body.Close()
	
	if resp.StatusCode != 200 {
		return nil, fmt.Errorf("[%s] 搜索请求返回状态码: %d", p.Name(), resp.StatusCode)
	}
	
	// 6. 解析搜索结果页面
	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("[%s] 解析搜索页面失败: %w", p.Name(), err)
	}
	
	// 7. 提取搜索结果
	var results []model.SearchResult
	
	doc.Find(".module-search-item").Each(func(i int, s *goquery.Selection) {
		result := p.parseSearchItem(s, keyword)
		if result.UniqueID != "" {
			results = append(results, result)
		}
	})
	
	// 8. 异步获取详情页信息
	enhancedResults := p.enhanceWithDetails(client, results)
	
	// 9. 关键词过滤
	return plugin.FilterResultsByKeyword(enhancedResults, keyword), nil
}

View on GitHub (pinned to beaa561337)