fish2018/pansou · error

[ ] 解析搜索页面失败

Error message

[%s] 解析搜索页面失败: %w

What it means

searchImpl wraps errors from goquery.NewDocumentFromReader, which parses the search response body as an HTML document. The response reached status 200 but goquery (built on golang.org/x/net/html) could not parse the body — the content is not valid HTML, is compressed/unreadable, or the body is truncated/empty due to an interrupted transfer.

Solutions

1. Capture the first bytes of the response body (Content-Type plus a snippet) when this error occurs to see what was actually returned.
2. Check the Content-Type header before parsing — skip or log when it is not text/html.
3. Inspect the raw body with curl to determine whether the site changed its markup or serves challenge pages on 200.
4. Check for mid-stream read errors (connection reset) in the wrapped error and rely on doRequestWithRetry for such transient failures.
5. Ensure the Transport isn't disabling automatic decompression (DisableCompression) while the server sends gzip.

Example fix

// before
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
    return nil, fmt.Errorf("[%s] 解析搜索页面失败: %w", p.Name(), err)
}
// after
if ct := resp.Header.Get("Content-Type"); !strings.Contains(ct, "text/html") {
    return nil, fmt.Errorf("[%s] 非HTML响应: %s", p.Name(), ct)
}
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
    return nil, fmt.Errorf("[%s] 解析搜索页面失败: %w", p.Name(), err)
}

// Validate the response is parseable HTML before goquery
ct := resp.Header.Get("Content-Type")
if !strings.Contains(ct, "text/html") {
    return nil, fmt.Errorf("unexpected content-type: %s", ct)
}
head, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
if !strings.Contains(string(head), "<") {
    return nil, fmt.Errorf("response body is not HTML")
}

// Go: wrap parse failure with body context for diagnosis
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
    return nil, fmt.Errorf("parse search page (status=%d, ct=%s): %w",
        resp.StatusCode, resp.Header.Get("Content-Type"), err)
}

When it happens

Trigger: NewDocumentFromReader returns an error when the response body contains malformed HTML that the html parser rejects, or when reading the body fails mid-stream (connection reset during body read, gzip/deflate corruption, chunked encoding truncated).

Common situations: The site returns a 200 page that is actually a JSON error, a CAPTCHA/challenge page with broken markup, or binary content; a proxy or antivirus mangles the body; the server closes the connection mid-response; custom Transport settings interfere with automatic gzip handling.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

• [ ] 解析搜索结果HTML失败
• [ ] 第 页HTML解析失败
• [ ] HTML解析失败
• [ ] 解析搜索页面失败
• [ ] HTML解析失败

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/0deb7a760b6c70aa. Report an issue: GitHub.

Appendix: source

Thrown at plugin/alupan/alupan.go:143

	if err != nil {
		return nil, fmt.Errorf("[%s] 创建请求失败: %w", p.Name(), err)
	}

	setCommonHeaders(req, "https://www.aliupan.com/")

	resp, err := p.doRequestWithRetry(req, client, searchMaxRetries)
	if err != nil {
		return nil, fmt.Errorf("[%s] 搜索请求失败: %w", p.Name(), err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("[%s] 搜索返回状态码: %d", p.Name(), resp.StatusCode)
	}

	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("[%s] 解析搜索页面失败: %w", p.Name(), err)
	}

	var (
		results []model.SearchResult
		wg      sync.WaitGroup
		mu      sync.Mutex
		sem     = make(chan struct{}, maxConcurrency)
	)

	doc.Find("article.excerpt").Each(func(_ int, item *goquery.Selection) {
		titleSel := item.Find("header h2 a")
		title := strings.TrimSpace(titleSel.Text())
		detailURL, ok := titleSel.Attr("href")
		if !ok || title == "" || detailURL == "" {
			return
		}

		articleID := extractArticleID(detailURL)

View on GitHub (pinned to beaa561337)