What causes "[%s] 第%d页解析失败: %w" in fish2018/pansou?

Log the first bytes of the response body and the Content-Type header to see what was actually returned.

fish2018/pansou · error

[ ] 第页解析失败

Error message

[%s] 第%d页解析失败: %w

What it means

After a successful response, fetchPage parses the (size-limited) body into a goquery document. If goquery.NewDocumentFromReader fails, the body was not valid parseable HTML (or an I/O error occurred while reading), and the error is wrapped with plugin name and page number.

Solutions

Log the first bytes of the response body and the Content-Type header to see what was actually returned.
Raise maxResponseSize if valid pages exceed the current cap and are being truncated mid-document.
Ensure the HTTP transport decodes gzip (check Accept-Encoding handling) before parsing.
Check status code before parsing so error payloads are not fed to goquery.

Example fix

// before
doc, err := goquery.NewDocumentFromReader(io.LimitReader(resp.Body, maxResponseSize))
if err != nil {
	return pageResult{}, fmt.Errorf("[%s] 第%d页解析失败: %w", p.Name(), page, err)
}
// after
ct := resp.Header.Get("Content-Type")
doc, err := goquery.NewDocumentFromReader(io.LimitReader(resp.Body, maxResponseSize))
if err != nil {
	return pageResult{}, fmt.Errorf("[%s] 第%d页解析失败 (status=%d content-type=%s): %w", p.Name(), page, resp.StatusCode, ct, err)
}

Defensive patterns

Strategy: validation

Validate before calling

ct := resp.Header.Get("Content-Type")
if !strings.Contains(ct, "text/html") {
	return fmt.Errorf("unexpected content-type %q; skip parsing", ct)
}

Type guard

func isHTMLResponse(resp *http.Response) bool { return strings.Contains(resp.Header.Get("Content-Type"), "text/html") }

Try / catch

doc, err := goquery.NewDocumentFromReader(io.LimitReader(resp.Body, maxResponseSize))
if err != nil {
	log.Printf("page %d unparseable (status=%d ct=%s)", page, resp.StatusCode, resp.Header.Get("Content-Type"))
	return pageResult{}, err
}

Prevention

Verify status 200 and HTML content type before parsing.
Keep maxResponseSize above typical page sizes to avoid mid-tag truncation.
Let the HTTP transport handle gzip decoding rather than parsing compressed bytes.

When it happens

Trigger: The response body, limited to maxResponseSize by io.LimitReader, is not parseable as HTML — e.g. it is a JSON error payload, gzip/charset mismatch, or truncated mid-tag by the size cap.

Common situations: Server returned JSON or plaintext error instead of HTML; response truncated exactly by maxResponseSize leaving broken markup the parser chokes on; Content-Encoding not transparently decoded by the transport.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/920d2be5f795f9d8. Report an issue: GitHub.

Appendix: source

Thrown at plugin/xiaoyu/xiaoyu.go:164

	requestURL := fmt.Sprintf(searchURL, page, url.PathEscape(keyword))
	ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, requestURL, nil)
	if err != nil {
		return pageResult{}, fmt.Errorf("[%s] 创建第%d页请求失败: %w", p.Name(), page, err)
	}
	setRequestHeaders(req)

	resp, err := doRequestWithRetry(client, req)
	if err != nil {
		return pageResult{}, fmt.Errorf("[%s] 第%d页搜索请求失败: %w", p.Name(), page, err)
	}
	defer resp.Body.Close()

	doc, err := goquery.NewDocumentFromReader(io.LimitReader(resp.Body, maxResponseSize))
	if err != nil {
		return pageResult{}, fmt.Errorf("[%s] 第%d页解析失败: %w", p.Name(), page, err)
	}

	return parsePage(doc), nil
}

func setRequestHeaders(req *http.Request) {
	req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
	req.Header.Set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Referer", baseURL+"/")
}

func doRequestWithRetry(client *http.Client, req *http.Request) (*http.Response, error) {
	var lastErr error
	for attempt := 0; attempt <= maxRetries; attempt++ {
		if attempt > 0 {
			time.Sleep(time.Duration(attempt) * 200 * time.Millisecond)

View on GitHub (pinned to beaa561337)

[ ] 第 页解析失败