{"record":{"id":"4de7e425895cede1","repo":"xpzouying/xiaohongshu-mcp","slug":"download-failed-with-status-d-for-url-s","errorCode":null,"errorMessage":"download failed with status %d for URL: %s","messagePattern":"download failed with status (.+?) for URL: (.+?)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/downloader/images.go","lineNumber":70,"sourceCode":"\n\t// 设置 User-Agent，模拟浏览器请求\n\treq.Header.Set(\"User-Agent\", \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36\")\n\n\t// 设置 Referer，使用图片 URL 的域名\n\tparsedURL, _ := url.Parse(imageURL)\n\tif parsedURL != nil {\n\t\treq.Header.Set(\"Referer\", fmt.Sprintf(\"%s://%s/\", parsedURL.Scheme, parsedURL.Host))\n\t}\n\n\t// 下载图片数据\n\tresp, err := d.httpClient.Do(req)\n\tif err != nil {\n\t\treturn \"\", errors.Wrapf(err, \"failed to download image from %s\", imageURL)\n\t}\n\tdefer resp.Body.Close()\n\n\tif resp.StatusCode != http.StatusOK {\n\t\treturn \"\", fmt.Errorf(\"download failed with status %d for URL: %s\", resp.StatusCode, imageURL)\n\t}\n\n\t// 读取图片数据\n\timageData, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\treturn \"\", errors.Wrap(err, \"failed to read image data\")\n\t}\n\n\t// 检测图片格式\n\tkind, err := filetype.Match(imageData)\n\tif err != nil {\n\t\treturn \"\", errors.Wrap(err, \"failed to detect file type\")\n\t}\n\n\tif !filetype.IsImage(imageData) {\n\t\treturn \"\", errors.New(\"downloaded file is not a valid image\")\n\t}\n","sourceCodeStart":52,"sourceCodeEnd":88,"githubUrl":"https://github.com/xpzouying/xiaohongshu-mcp/blob/332d196854a9eac0d2b8c2c0e3d0cc43139d724c/pkg/downloader/images.go#L52-L88","documentation":"DownloadImage fetches a single image via HTTP and returns its bytes/local path. This error is thrown when the response status is not 200 OK, so the body is not trusted as image data and the download is abandoned.","triggerScenarios":"DownloadImages or ProcessImages calls d.DownloadImage(imageURL); the HTTP request succeeds but the server responds with e.g. 404 (image moved/deleted), 403 (hotlink protection / missing UA or Referer — note the UA/Referer headers this client sends), 429 (rate limited), or 5xx.","commonSituations":"Hotlink protection rejecting requests without proper Referer/User-Agent; images behind expiring signed URLs (S3/CDN token expired -> 403); rate limiting (429) when scraping many images from one host; image removed after the page HTML was parsed (stale URL); geo-blocked CDN.","solutions":["Log the actual status code and URL; for 403 check whether the site requires the UA/Referer headers this downloader already sends, and whether the signed URL expired","For 429, add throttling/delay between requests and retry with exponential backoff","Re-parse the page to get fresh image URLs if they are expiring or the image was moved (404)","Verify the URL scheme is correct (http vs https, no relative path passed through unchanged)","For persistent 5xx, retry later or skip the image and continue the batch (DownloadImages already collects per-URL errors)"],"exampleFix":"// before\nif resp.StatusCode != http.StatusOK {\n\treturn \"\", fmt.Errorf(\"download failed with status %d for URL: %s\", resp.StatusCode, imageURL)\n}\n// after — retry on transient statuses with backoff\nif resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {\n\ttime.Sleep(backoff)\n\treturn d.DownloadImage(imageURL) // retry\n}\nif resp.StatusCode != http.StatusOK {\n\treturn \"\", fmt.Errorf(\"download failed with status %d for URL: %s\", resp.StatusCode, imageURL)\n}","handlingStrategy":"retry","validationCode":"resp, err := http.Head(imageURL)\nif err != nil || resp.StatusCode != http.StatusOK {\n\t// 图片不可达（403/404/429），跳过或换源\n}","typeGuard":null,"tryCatchPattern":"path, err := d.DownloadImage(imageURL)\nif err != nil {\n\tvar statusErr interface{ Error() string }\n\t_ = statusErr\n\tif strings.Contains(err.Error(), \"status 429\") || strings.Contains(err.Error(), \"status 5\") {\n\t\ttime.Sleep(2 * time.Second)\n\t\tpath, err = d.DownloadImage(imageURL) // 退避重试一次\n\t}\n\tif err != nil { log.Printf(\"skip %s: %v\", imageURL, err) }\n}","preventionTips":["Send proper User-Agent/Referer headers to bypass hotlink protection","Throttle requests per host to avoid 429 rate limiting","Re-scrape page URLs if they are signed and expiring","Validate the URL is absolute and scheme-correct before downloading"],"tags":["network","http","image-download","scraping"],"backgroundTag":"http-non-200-response","analyzedSha":"332d196854a9eac0d2b8c2c0e3d0cc43139d724c","analyzedAt":"2026-09-05T22:22:55.988Z","contentChangedAt":"2026-09-05T22:22:55.988Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}