fish2018/pansou · error

[ ] 解析token页面HTML失败

Error message

[%s] 解析token页面HTML失败: %w

What it means

getToken wraps the error from goquery.NewDocumentFromReader when the token page body cannot be parsed as HTML. Since the body was already streamed from resp.Body, this usually indicates the response was truncated, empty, or binary/compressed content that goquery (html.Parse) cannot handle.

Solutions

1. Capture the raw body first (io.ReadAll) and check it is non-empty and looks like HTML before parsing
2. Ensure the http.Client Transport has DecompressBody/Accept-Encoding handled (DisableCompression=false) so gzip bodies are decoded
3. Log a body prefix on parse failure to see what the server returned
4. If the site changed its page structure, verify the token page is still HTML and update scraping accordingly

Example fix

// before
doc, err := goquery.NewDocumentFromReader(resp.Body)
// after
raw, rerr := io.ReadAll(resp.Body)
if rerr != nil {
    return "", fmt.Errorf("[%s] 读取token页面失败: %w", p.Name(), rerr)
}
doc, err := goquery.NewDocumentFromReader(bytes.NewReader(raw))

raw, _ := io.ReadAll(resp.Body)
if len(raw) == 0 || !bytes.Contains(bytes.ToLower(raw[:min(512, len(raw))]), []byte("<")) {
    return fmt.Errorf("response is not HTML")
}

doc, err := goquery.NewDocumentFromReader(bytes.NewReader(raw))
if err != nil {
    log.Printf("token page HTML parse failed (%d bytes): %q", len(raw), raw[:min(200, len(raw))])
    return errTokenPageUnparseable
}

When it happens

Trigger: goquery.NewDocumentFromReader(resp.Body) returns err — malformed/truncated HTML, empty body after a 200 response, or non-HTML content (gzip not decompressed, binary data) on the token page.

Common situations: Anti-bot interstitial serving broken HTML with status 200; proxy mangling the response; server returning compressed body without the client transparently decompressing it; Content-Sniffing mismatch.

Related errors

• [ ] 解析搜索页面失败
• [ ] HTML解析失败
• [ ] 解析搜索结果HTML失败
• [ ] 解析搜索页面失败
• [ ] 解析搜索结果失败

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/951f2c2d2824e25e. Report an issue: GitHub.

Appendix: source

Thrown at plugin/xys/xys.go:174

	req.Header.Set("Connection", "keep-alive")
	req.Header.Set("Upgrade-Insecure-Requests", "1")
	req.Header.Set("Cache-Control", "max-age=0")
	req.Header.Set("Referer", BaseURL+"/")

	resp, err := p.doRequestWithRetry(req, client)
	if err != nil {
		return "", fmt.Errorf("[%s] token请求失败: %w", p.Name(), err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != 200 {
		return "", fmt.Errorf("[%s] token请求HTTP状态错误: %d", p.Name(), resp.StatusCode)
	}

	// 解析HTML提取token
	doc, err := goquery.NewDocumentFromReader(resp.Body)
	if err != nil {
		return "", fmt.Errorf("[%s] 解析token页面HTML失败: %w", p.Name(), err)
	}

	// 查找script标签中的DToken定义
	var token string
	doc.Find("script").Each(func(i int, s *goquery.Selection) {
		scriptContent := s.Text()
		if strings.Contains(scriptContent, "DToken") {
			// 使用正则表达式提取token
			re := regexp.MustCompile(`const\s+DToken\s*=\s*"([^"]+)"`)
			matches := re.FindStringSubmatch(scriptContent)
			if len(matches) > 1 {
				token = matches[1]
				if p.debugMode {
					log.Printf("[XYS] 从script中提取到token: %s", token[:10]+"...")
				}
			}
		}
	})

View on GitHub (pinned to beaa561337)