fish2018/pansou · error

解析HTML失败

Error message

解析HTML失败: %w

What it means

goquery.NewDocumentFromReader failed to parse the response body as HTML. goquery (via x/net/html) rarely errors on arbitrary text, so this usually means the body was not decodable content — e.g. compressed/gzip body read incorrectly, truncated response, or binary data.

Solutions

  1. Check the Content-Encoding/Content-Type headers; ensure gzip is handled by the transport (default Go transport handles it).
  2. Log the first bytes of the body on parse failure to see what was actually received.
  3. Retry the request if the body was truncated by a network error.
  4. Update goquery/golang.org/x/net to the latest version for parser fixes.
  5. If charset issues, decode with golang.org/x/net/html/charset before parsing.

Example fix

// before
doc, err := goquery.NewDocumentFromReader(resp2.Body)
if err != nil {
    return nil, fmt.Errorf("解析HTML失败: %w", err)
}
// after
doc, err := goquery.NewDocumentFromReader(resp2.Body)
if err != nil {
    return nil, fmt.Errorf("解析HTML失败: %w", err) // log Content-Type/Encoding and body head for diagnosis
}
Defensive patterns

Strategy: try-catch

Validate before calling

// Go: verify content type before parsing
ct := resp2.Header.Get("Content-Type")
if !strings.Contains(ct, "text/html") {
    return nil, fmt.Errorf("unexpected content type: %s", ct)
}

Try / catch

results, err := plugin.Search(keyword, page)
if err != nil && strings.Contains(err.Error(), "解析HTML失败") {
    log.Printf("HTML parse failed, possibly truncated or non-HTML body: %v", err)
    // retry once; then alert that the scraper may need updating
    return err
}

Prevention

When it happens

Trigger: goquery.NewDocumentFromReader(resp2.Body) returns a parse error: malformed/truncated HTML stream, charset issues producing invalid tokens, or a non-HTML body (binary) when the server mislabeled content type.

Common situations: Server returned gzipped content with an exotic configuration; connection cut mid-body causing truncated markup; custom transport not handling Content-Encoding; upstream returning JSON/binary on this endpoint after a site change.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/1d5cbc712fe2b5b4. Report an issue: GitHub.

Appendix: source

Thrown at plugin/panwiki/panwiki.go:252

		return nil, fmt.Errorf("创建搜索请求失败: %w", err)
	}
	
	p.setRequestHeaders(req2)
	
	resp2, err := client.Do(req2)
	if err != nil {
		return nil, fmt.Errorf("搜索请求失败: %w", err)
	}
	defer resp2.Body.Close()
	
	if resp2.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("搜索请求返回状态码: %d", resp2.StatusCode)
	}
	
	// 解析搜索结果
	doc, err := goquery.NewDocumentFromReader(resp2.Body)
	if err != nil {
		return nil, fmt.Errorf("解析HTML失败: %w", err)
	}
	
	return p.extractSearchResults(doc), nil
}

// setRequestHeaders 设置请求头
func (p *PanwikiPlugin) setRequestHeaders(req *http.Request) {
	req.Header.Set("User-Agent", UserAgent)
	req.Header.Set("Referer", p.currentBaseURL+"/")
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
	req.Header.Set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8")
	req.Header.Set("Cache-Control", "no-cache")
	req.Header.Set("Pragma", "no-cache")
}

// extractSearchResults 提取搜索结果
func (p *PanwikiPlugin) extractSearchResults(doc *goquery.Document) []model.SearchResult {
	var results []model.SearchResult

View on GitHub (pinned to beaa561337)