What causes "[%s] HTML解析失败: %w" in fish2018/pansou?

Check the Content-Type and Content-Encoding headers of the response before parsing.

fish2018/pansou · error

[ ] HTML解析失败

Error message

[%s] HTML解析失败: %w

What it means

CldiPlugin.searchPage wraps goquery.NewDocumentFromReader failures with the plugin name. This is uncommon because Go's HTML parser is lenient, but it fires when the reader itself fails (nil/failed reader) or the input cannot be parsed at all — for example binary/compressed content delivered without decompression.

Solutions

Check the Content-Type and Content-Encoding headers of the response before parsing.
If content is compressed, ensure the transport decompresses (DisableCompression=false) or decompress manually with gzip.NewReader.
Log the first bytes of body to see whether it is actually HTML.
Verify the endpoint still returns an HTML search page and not an API/binary response.

Example fix

// before
doc, err := goquery.NewDocumentFromReader(strings.NewReader(string(body)))
if err != nil {
    return nil, fmt.Errorf("[%s] HTML解析失败: %w", p.Name(), err)
}
// after
ct := resp.Header.Get("Content-Type")
if !strings.Contains(ct, "text/html") {
    return nil, fmt.Errorf("[%s] 非HTML响应(Content-Type=%s), 前200字节: %q", p.Name(), ct, body[:min(len(body),200)])
}
doc, err := goquery.NewDocumentFromReader(strings.NewReader(string(body)))
if err != nil {
    return nil, fmt.Errorf("[%s] HTML解析失败: %w", p.Name(), err)
}

Defensive patterns

Strategy: validation

Validate before calling

ct := resp.Header.Get("Content-Type")
enc := resp.Header.Get("Content-Encoding")
if enc != "" && enc != "identity" {
    return fmt.Errorf("未处理的压缩编码: %s", enc)
}
if !strings.Contains(ct, "text/html") {
    return fmt.Errorf("非HTML响应: %s", ct)
}

Type guard

func looksLikeHTML(body []byte) bool {
    s := strings.TrimSpace(strings.ToLower(string(body[:min(len(body),512)])));
    return strings.HasPrefix(s, "<!doctype html") || strings.HasPrefix(s, "<html")
}

Try / catch

results, err := plugin.Search(ctx, keyword)
if err != nil {
    if strings.Contains(err.Error(), "HTML解析失败") {
        log.Printf("CLDI返回了不可解析内容: %v", err)
        // skip plugin or switch to alternate source
    }
}

Prevention

Verify Content-Type/Content-Encoding before parsing the body
Let net/http decompress gzip automatically (don't set Accept-Encoding manually unless handling it)
Spot-check response bytes in tests with recorded fixtures
Update extraction selectors when the site's HTML changes

When it happens

Trigger: goquery.NewDocumentFromReader(strings.NewReader(string(body))) returns err in searchPage — most plausibly when body is gzip-compressed bytes (content-encoding not handled) or the read errored.

Common situations: Server returning gzip/brotli content while the client didn't send Accept-Encoding or didn't decompress, a binary CAPTCHA/challenge page, or corrupted transfer.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/62cbb9bb55365087. Report an issue: GitHub.

Appendix: source

Thrown at plugin/cldi/cldi.go:163

		return nil, fmt.Errorf("[%s] 搜索请求失败: %w", p.Name(), err)
	}
	defer resp.Body.Close()

	// 检查状态码
	if resp.StatusCode != 200 {
		return nil, fmt.Errorf("[%s] 请求返回状态码: %d", p.Name(), resp.StatusCode)
	}

	// 读取响应
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, fmt.Errorf("[%s] 读取响应失败: %w", p.Name(), err)
	}

	// 解析HTML
	doc, err := goquery.NewDocumentFromReader(strings.NewReader(string(body)))
	if err != nil {
		return nil, fmt.Errorf("[%s] HTML解析失败: %w", p.Name(), err)
	}

	// 提取搜索结果
	return p.extractSearchResults(doc), nil
}

// setRequestHeaders 设置请求头
func (p *CldiPlugin) setRequestHeaders(req *http.Request) {
	req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
	req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
	req.Header.Set("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8")
	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Cache-Control", "no-cache")
	req.Header.Set("Pragma", "no-cache")
	req.Header.Set("Referer", baseURL+"/")
}

// doRequestWithRetry 带重试机制的HTTP请求

View on GitHub (pinned to beaa561337)