What causes "[%s] HTML解析失败: %w" in fish2018/pansou?

Check the wrapped parse error; if it mentions gzip/zlib, let net/http auto-decompress by not setting Accept-Encoding manually

fish2018/pansou · error

[ ] HTML解析失败

Error message

[%s] HTML解析失败: %w

What it means

goquery.NewDocumentFromReader failed to parse the kkv search response body as HTML. This happens when the 200 response is not valid parseable HTML — e.g. a compressed/binary body, truncated response, or a challenge page that breaks the parser.

Solutions

Check the wrapped parse error; if it mentions gzip/zlib, let net/http auto-decompress by not setting Accept-Encoding manually
Dump the first bytes of resp.Body to confirm the 200 response is actually HTML and not empty or a challenge page
Check the proxy/CDN in front of the scraper isn't corrupting or truncating bodies
Retry the request if the body was truncated by a flaky connection (doRequestWithRetry already retries transport errors, but not short bodies)

Example fix

// before
req.Header.Set("Accept-Encoding", "gzip, deflate")
// after
// omit Accept-Encoding so net/http transparently decompresses for goquery

Defensive patterns

Strategy: validation

Validate before calling

head, _ := client.Head(searchURL)
ct := head.Header.Get("Content-Type")
if !strings.Contains(ct, "text/html") {
    // expect HTML parse failure; skip source
}

Type guard

func looksLikeHTML(b []byte) bool {
    s := strings.TrimSpace(strings.ToLower(string(b[:min(len(b), 512)])))
    return strings.HasPrefix(s, "<!doctype html") || strings.HasPrefix(s, "<html")
}

Try / catch

if err != nil {
    if strings.Contains(err.Error(), "HTML解析失败") {
        log.Printf("non-HTML 200 body from source, skipping: %v", err)
        return nil
    }
    return err
}

Prevention

Never set Accept-Encoding manually; let net/http decompress
Check Content-Type of upstream responses before parsing
Beware proxies/CDNs that return empty or challenge 200 bodies
Sniff the first bytes when debugging parse errors

When it happens

Trigger: goquery.NewDocumentFromReader(resp.Body) returns err after a 200 status: body is gzip-encoded but not auto-decompressed, connection truncated mid-body, or the body is JSON/empty instead of HTML.

Common situations: A custom http.Client without automatic gzip handling receiving encoded bytes, a proxy returning an error page or empty 200 body, the site serving a Cloudflare challenge that goquery chokes on.

Related errors

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/b5a7a8c3266c290b. Report an issue: GitHub.

Appendix: source

Thrown at plugin/kkv/kkv.go:142

	}
	
	p.setHeaders(req, baseURL)
	
	resp, err := p.doRequestWithRetry(req, client)
	if err != nil {
		return nil, fmt.Errorf("[%s] 搜索请求失败: %w", p.Name(), err)
	}
	defer resp.Body.Close()
	
	debugPrintf("📡 HTTP状态码: %d\n", resp.StatusCode)
	
	if resp.StatusCode != 200 {
		return nil, fmt.Errorf("[%s] 请求返回状态码: %d", p.Name(), resp.StatusCode)
	}
	
	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("[%s] HTML解析失败: %w", p.Name(), err)
	}
	
	var items []searchItem
	doc.Find("article.post").Each(func(i int, s *goquery.Selection) {
		link := s.Find(".entry-header h2.entry-title a")
		href, exists := link.Attr("href")
		if !exists {
			debugPrintf("⚠️ 第%d个结果没有href属性\n", i+1)
			return
		}
		
		title := strings.TrimSpace(link.Text())
		if title == "" {
			debugPrintf("⚠️ 第%d个结果标题为空\n", i+1)
			return
		}
		
		re := regexp.MustCompile(`\?p=(\d+)`)

View on GitHub (pinned to beaa561337)