{"record":{"id":"9b8cd6fad5393c4f","repo":"fish2018/pansou","slug":"gzip-reader-w-9b8cd6","errorCode":null,"errorMessage":"创建gzip reader失败: %w","messagePattern":"创建gzip reader失败: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"plugin/xiaozhang/xiaozhang.go","lineNumber":168,"sourceCode":"\tif resp.StatusCode != http.StatusOK {\n\t\treturn nil, fmt.Errorf(\"搜索响应状态码异常: %d\", resp.StatusCode)\n\t}\n\t\n\t// 处理响应体（可能是gzip压缩的）\n\tvar reader io.Reader = resp.Body\n\t\n\t// 检查Content-Encoding\n\tcontentEncoding := resp.Header.Get(\"Content-Encoding\")\n\tif p.debugMode {\n\t\tlog.Printf(\"[Xiaozhang] Content-Encoding: %s\", contentEncoding)\n\t\tlog.Printf(\"[Xiaozhang] Content-Type: %s\", resp.Header.Get(\"Content-Type\"))\n\t}\n\t\n\t// 如果是gzip压缩，手动解压\n\tif contentEncoding == \"gzip\" {\n\t\tgzReader, err := gzip.NewReader(resp.Body)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"创建gzip reader失败: %w\", err)\n\t\t}\n\t\tdefer gzReader.Close()\n\t\treader = gzReader\n\t}\n\t\n\t// 解析HTML\n\tdoc, err := goquery.NewDocumentFromReader(reader)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"解析HTML失败: %w\", err)\n\t}\n\t\n\t// 提取搜索结果\n\tresults := p.extractSearchResults(doc, keyword)\n\t\n\tif p.debugMode {\n\t\tlog.Printf(\"[Xiaozhang] 找到 %d 个搜索结果\", len(results))\n\t}\n\t","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/fish2018/pansou/blob/beaa56133755a548ebc51b090b3816e2ae044aa6/plugin/xiaozhang/xiaozhang.go#L150-L186","documentation":"The xiaozhang plugin disables the http.Transport's automatic compression (DisableCompression: true) and manually decompresses when Content-Encoding is gzip. This error is thrown when gzip.NewReader(resp.Body) fails, meaning the response body is not actually a valid gzip stream despite the server claiming Content-Encoding: gzip. gzip.NewReader reads the header lazily, so the first bytes must be the gzip magic number 0x1f 0x8b.","triggerScenarios":"The search endpoint returns a Content-Encoding: gzip header but the body is plain text/HTML, an anti-bot challenge page, or a corrupted/truncated stream — so gzip.NewReader fails to parse the header.","commonSituations":"A WAF or CDN strips or rewrites the body but keeps the encoding header; a middlebox (corporate proxy) decompresses the response without removing the Content-Encoding header; the site misconfigures content compression; or the response is a cached error page.","solutions":["Log the first bytes of resp.Body (hex) to confirm whether it is actually gzip (should start with 1f 8b).","Handle gzip.ErrHeader specifically — it means the body is not gzip at all; fall back to parsing it as plain HTML.","Detect corruption with http.ErrBodyReadAfterClose / unexpected EOF and retry the request.","Consider re-enabling automatic compression (DisableCompression: false) and letting net/http decompress, removing manual gzip handling.","Add a sniffing helper that only wraps in gzip.NewReader when the body starts with the gzip magic bytes."],"exampleFix":"// before\nif contentEncoding == \"gzip\" {\n    gzReader, err := gzip.NewReader(resp.Body)\n    if err != nil {\n        return nil, fmt.Errorf(\"创建gzip reader失败: %w\", err)\n    }\n    defer gzReader.Close()\n    reader = gzReader\n}\n// after\nif contentEncoding == \"gzip\" {\n    br := bufio.NewReader(resp.Body)\n    magic, _ := br.Peek(2)\n    if len(magic) == 2 && magic[0] == 0x1f && magic[1] == 0x8b {\n        gzReader, err := gzip.NewReader(br)\n        if err != nil {\n            return nil, fmt.Errorf(\"创建gzip reader失败: %w\", err)\n        }\n        defer gzReader.Close()\n        reader = gzReader\n    } else {\n        reader = br // server lied about gzip; parse as plain HTML\n    }\n}","handlingStrategy":"fallback","validationCode":"// Go: verify the advertised encoding against the actual body magic bytes\nif resp.Header.Get(\"Content-Encoding\") == \"gzip\" {\n    br := bufio.NewReader(resp.Body)\n    magic, _ := br.Peek(2)\n    if !(len(magic) == 2 && magic[0] == 0x1f && magic[1] == 0x8b) {\n        return fmt.Errorf(\"server claims gzip but body is not gzip\")\n    }\n}","typeGuard":"// Go: safe gzip sniffing helper\nfunc isGzip(r io.Reader) (bool, io.Reader, error) {\n    br := bufio.NewReader(r)\n    magic, err := br.Peek(2)\n    if err != nil && err != io.EOF {\n        return false, br, err\n    }\n    return len(magic) == 2 && magic[0] == 0x1f && magic[1] == 0x8b, br, nil\n}","tryCatchPattern":"reader, err := wrapMaybeGzip(resp.Body, resp.Header.Get(\"Content-Encoding\"))\nif err != nil {\n    var gzErr *gzip.HeaderError\n    if errors.As(err, &gzErr) {\n        log.Printf(\"body not gzip despite header; falling back to plain read\")\n        reader = resp.Body // fallback: parse as plain HTML\n    } else {\n        return nil, err\n    }\n}","preventionTips":["Disable transparent middleboxes/proxies that rewrite bodies without fixing headers, or sniff magic bytes instead of trusting Content-Encoding.","Prefer letting net/http auto-decompress (DisableCompression: false) unless you have a specific reason to decompress manually.","Peek at the first 2 bytes before gzip.NewReader to confirm the gzip magic number.","Handle gzip.ErrHeader explicitly with a plain-HTML fallback path.","Log response headers when debugging scraping plugins to spot encoding mismatches quickly."],"tags":["gzip","compression","http-response","decompression"],"backgroundTag":"invalid-json-response","analyzedSha":"beaa56133755a548ebc51b090b3816e2ae044aa6","analyzedAt":"2026-09-07T00:31:18.025Z","contentChangedAt":"2026-09-07T00:31:18.025Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}