What causes "[%s] 搜索请求返回状态码: %d" in fish2018/pansou?

Log the status code and read the body on 403/503 to identify Cloudflare or captcha challenges.

fish2018/pansou · error

[ ] 搜索请求返回状态码

Error message

[%s] 搜索请求返回状态码: %d

What it means

After doRequestWithRetry succeeds, searchImpl checks resp.StatusCode != 200 and returns this formatted message with the numeric status. The site responded, but not with an OK status — for a scraped WordPress-style search on ahhhhfs.com this usually means anti-bot/WAF blocking (403), rate limiting (429), a moved path (404), or a server error (5xx).

Solutions

Log the status code and read the body on 403/503 to identify Cloudflare or captcha challenges.
Replay the request with curl using the same headers to confirm whether it's header-based blocking.
Refresh the request headers (current browser User-Agent, cookies) used to dodge anti-bot checks.
Add throttling/backoff between searches to avoid 429; retry after the Retry-After interval when present.
Update the searchURL if the site changed its search endpoint structure.

Example fix

// before
if resp.StatusCode != 200 {
    return nil, fmt.Errorf("[%s] 搜索请求返回状态码: %d", p.Name(), resp.StatusCode)
}
// after
if resp.StatusCode != 200 {
    snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
    return nil, fmt.Errorf("[%s] 搜索请求返回状态码: %d, body: %s", p.Name(), resp.StatusCode, snippet)
}

Defensive patterns

Strategy: retry

Try / catch

results, err := p.Search(ctx, keyword)
var codeErr *statusCodeError // 若封装了状态码
if err != nil && strings.Contains(err.Error(), "429") {
    time.Sleep(retryAfter) // 遵循 Retry-After 后重试
    results, err = p.Search(ctx, keyword)
}

Prevention

Rate-limit searches and add jittered backoff to avoid 429/403.
Keep anti-bot headers (current User-Agent, cookies) fresh.
On repeated 4xx, check the site in a browser for new WAF/captcha protection.
Update the search URL when the site changes structure.

When it happens

Trigger: The GET to the ahhhhfs search URL returns 301/302 (redirect not resolved to 200), 403 (Cloudflare/WAF rejects the spoofed browser headers), 404 (search URL structure changed), 429 (too many requests), or 5xx while the site is degraded.

Common situations: Scraping too aggressively triggers rate limiting; the site enabled stricter bot protection that detects the hardcoded Chrome 120 User-Agent; the site changed its search path; temporary site outage (502/503).

Related errors

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/356629e71a1d3a53. Report an issue: GitHub.

Appendix: source

Thrown at plugin/ahhhhfs/ahhhhfs.go:186

	
	// 4. 设置完整的请求头（避免反爬虫）
	req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
	req.Header.Set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Upgrade-Insecure-Requests", "1")
	req.Header.Set("Cache-Control", "max-age=0")
	req.Header.Set("Referer", "https://www.ahhhhfs.com/")
	
	// 5. 发送请求（带重试机制）
	resp, err := p.doRequestWithRetry(req, client)
	if err != nil {
		return nil, fmt.Errorf("[%s] 搜索请求失败: %w", p.Name(), err)
	}
	defer resp.Body.Close()
	
	if resp.StatusCode != 200 {
		return nil, fmt.Errorf("[%s] 搜索请求返回状态码: %d", p.Name(), resp.StatusCode)
	}
	
	// 6. 解析搜索结果页面
	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("[%s] 解析搜索页面失败: %w", p.Name(), err)
	}
	
	// 7. 提取搜索结果
	var results []model.SearchResult
	var wg sync.WaitGroup
	var mu sync.Mutex
	semaphore := make(chan struct{}, MaxConcurrency)
	
	doc.Find("article.post-item.item-list").Each(func(i int, s *goquery.Selection) {
		// 解析基本信息
		titleElem := s.Find(".entry-title a")
		title := strings.TrimSpace(titleElem.Text())

View on GitHub (pinned to beaa561337)