fish2018/pansou · error
parse web search page failed
Error message
parse web search page failed: %w
What it means
Once a 200 response is received, searchWeb parses it with goquery. If the body cannot be turned into an HTML document, 'parse web search page failed' is returned. It indicates the 200 response body was not valid/readable HTML for the panso search page.
Solutions
- Dump the first bytes of the body and Content-Type to confirm what was actually returned.
- Retry the request; intermittent truncation is common under load.
- Verify setSousouWebHeaders includes Accept/Accept-Encoding that yield plain HTML.
- If the page now requires JavaScript, switch to a headless-browser fetch for this source.
Example fix
// before
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
return nil, fmt.Errorf("parse web search page failed: %w", err)
}
// after
body, _ := io.ReadAll(resp.Body)
if len(bytes.TrimSpace(body)) == 0 {
return nil, fmt.Errorf("parse web search page failed: empty body")
}
doc, err := goquery.NewDocumentFromReader(bytes.NewReader(body)) Defensive patterns
Strategy: fallback
Try / catch
results, err := plugin.Search(keyword)
if err != nil && strings.Contains(err.Error(), "parse web search page failed") {
// capture body sample for debugging, retry once, then fall back to another source
} Prevention
- Read the body fully before parsing so failures are diagnosable
- Verify Accept/Accept-Encoding headers yield plain HTML
- Detect captcha/challenge pages explicitly before parsing
- Alert on sudden parse-failure increases (site redesign signal)
When it happens
Trigger: goquery.NewDocumentFromReader(resp.Body) errors inside searchWeb after a successful 200 response — truncated body, encoding problems, or non-HTML content served with 200.
Common situations: Anti-bot served a 200 with a JS-challenge page or empty body; transfer cut off; charset/encoding mismatch breaking the parser.
Related errors
AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07).
Data as JSON: /api/errors/fd05b7a8f14b7600.
Report an issue: GitHub.
Appendix: source
Thrown at plugin/sousou/sousou.go:192
searchURL := SousouWebURL + "?q=" + url.QueryEscape(strings.TrimSpace(keyword))
req, err := http.NewRequestWithContext(ctx, http.MethodGet, searchURL, nil)
if err != nil {
return nil, fmt.Errorf("create web search request failed: %w", err)
}
setSousouWebHeaders(req, "https://www.panso.vip/")
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("web search request failed: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("web search returned status %d", resp.StatusCode)
}
doc, err := goquery.NewDocumentFromReader(resp.Body)
if err != nil {
return nil, fmt.Errorf("parse web search page failed: %w", err)
}
items := make([]pansoSearchItem, 0, 20)
doc.Find("div.search-item").Each(func(_ int, item *goquery.Selection) {
anchor := item.Find("a.search-item-title[href]").First()
href := strings.TrimSpace(anchor.AttrOr("href", ""))
if href == "" {
return
}
items = append(items, pansoSearchItem{
DocURL: absolutePansoURL(href),
Title: strings.TrimSpace(anchor.Text()),
Content: strings.TrimSpace(item.Find(".search-item-info").Text()),
DiskType: strings.TrimSpace(item.Find(".search-item-logo").AttrOr("alt", "")),
Datetime: parsePansoDatetime(item.Find(".search-item-meta-item").Text()),
Password: parsePansoPassword(item),
})
})View on GitHub (pinned to beaa561337)