fish2018/pansou · error

创建gzip reader失败

Error message

创建gzip reader失败: %w

What it means

The xiaozhang plugin disables the http.Transport's automatic compression (DisableCompression: true) and manually decompresses when Content-Encoding is gzip. This error is thrown when gzip.NewReader(resp.Body) fails, meaning the response body is not actually a valid gzip stream despite the server claiming Content-Encoding: gzip. gzip.NewReader reads the header lazily, so the first bytes must be the gzip magic number 0x1f 0x8b.

Solutions

1. Log the first bytes of resp.Body (hex) to confirm whether it is actually gzip (should start with 1f 8b).
2. Handle gzip.ErrHeader specifically — it means the body is not gzip at all; fall back to parsing it as plain HTML.
3. Detect corruption with http.ErrBodyReadAfterClose / unexpected EOF and retry the request.
4. Consider re-enabling automatic compression (DisableCompression: false) and letting net/http decompress, removing manual gzip handling.
5. Add a sniffing helper that only wraps in gzip.NewReader when the body starts with the gzip magic bytes.

Example fix

// before
if contentEncoding == "gzip" {
    gzReader, err := gzip.NewReader(resp.Body)
    if err != nil {
        return nil, fmt.Errorf("创建gzip reader失败: %w", err)
    }
    defer gzReader.Close()
    reader = gzReader
}
// after
if contentEncoding == "gzip" {
    br := bufio.NewReader(resp.Body)
    magic, _ := br.Peek(2)
    if len(magic) == 2 && magic[0] == 0x1f && magic[1] == 0x8b {
        gzReader, err := gzip.NewReader(br)
        if err != nil {
            return nil, fmt.Errorf("创建gzip reader失败: %w", err)
        }
        defer gzReader.Close()
        reader = gzReader
    } else {
        reader = br // server lied about gzip; parse as plain HTML
    }
}

// Go: verify the advertised encoding against the actual body magic bytes
if resp.Header.Get("Content-Encoding") == "gzip" {
    br := bufio.NewReader(resp.Body)
    magic, _ := br.Peek(2)
    if !(len(magic) == 2 && magic[0] == 0x1f && magic[1] == 0x8b) {
        return fmt.Errorf("server claims gzip but body is not gzip")
    }
}

// Go: safe gzip sniffing helper
func isGzip(r io.Reader) (bool, io.Reader, error) {
    br := bufio.NewReader(r)
    magic, err := br.Peek(2)
    if err != nil && err != io.EOF {
        return false, br, err
    }
    return len(magic) == 2 && magic[0] == 0x1f && magic[1] == 0x8b, br, nil
}

reader, err := wrapMaybeGzip(resp.Body, resp.Header.Get("Content-Encoding"))
if err != nil {
    var gzErr *gzip.HeaderError
    if errors.As(err, &gzErr) {
        log.Printf("body not gzip despite header; falling back to plain read")
        reader = resp.Body // fallback: parse as plain HTML
    } else {
        return nil, err
    }
}

When it happens

Trigger: The search endpoint returns a Content-Encoding: gzip header but the body is plain text/HTML, an anti-bot challenge page, or a corrupted/truncated stream — so gzip.NewReader fails to parse the header.

Common situations: A WAF or CDN strips or rewrites the body but keeps the encoding header; a middlebox (corporate proxy) decompresses the response without removing the Content-Encoding header; the site misconfigures content compression; or the response is a cached error page.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

• 创建gzip reader失败
• 创建gzip reader失败
• 创建gzip reader失败
• read response body failed
• decode response failed

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/9b8cd6fad5393c4f. Report an issue: GitHub.

Appendix: source

Thrown at plugin/xiaozhang/xiaozhang.go:168

	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("搜索响应状态码异常: %d", resp.StatusCode)
	}
	
	// 处理响应体（可能是gzip压缩的）
	var reader io.Reader = resp.Body
	
	// 检查Content-Encoding
	contentEncoding := resp.Header.Get("Content-Encoding")
	if p.debugMode {
		log.Printf("[Xiaozhang] Content-Encoding: %s", contentEncoding)
		log.Printf("[Xiaozhang] Content-Type: %s", resp.Header.Get("Content-Type"))
	}
	
	// 如果是gzip压缩，手动解压
	if contentEncoding == "gzip" {
		gzReader, err := gzip.NewReader(resp.Body)
		if err != nil {
			return nil, fmt.Errorf("创建gzip reader失败: %w", err)
		}
		defer gzReader.Close()
		reader = gzReader
	}
	
	// 解析HTML
	doc, err := goquery.NewDocumentFromReader(reader)
	if err != nil {
		return nil, fmt.Errorf("解析HTML失败: %w", err)
	}
	
	// 提取搜索结果
	results := p.extractSearchResults(doc, keyword)
	
	if p.debugMode {
		log.Printf("[Xiaozhang] 找到 %d 个搜索结果", len(results))
	}
	

View on GitHub (pinned to beaa561337)