xpzouying/xiaohongshu-mcp · error

download failed with status %d for URL: %s

Error message

download failed with status %d for URL: %s

What it means

DownloadImage fetches a single image via HTTP and returns its bytes/local path. This error is thrown when the response status is not 200 OK, so the body is not trusted as image data and the download is abandoned.

Source

Thrown at pkg/downloader/images.go:70

	// 设置 User-Agent，模拟浏览器请求
	req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")

	// 设置 Referer，使用图片 URL 的域名
	parsedURL, _ := url.Parse(imageURL)
	if parsedURL != nil {
		req.Header.Set("Referer", fmt.Sprintf("%s://%s/", parsedURL.Scheme, parsedURL.Host))
	}

	// 下载图片数据
	resp, err := d.httpClient.Do(req)
	if err != nil {
		return "", errors.Wrapf(err, "failed to download image from %s", imageURL)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return "", fmt.Errorf("download failed with status %d for URL: %s", resp.StatusCode, imageURL)
	}

	// 读取图片数据
	imageData, err := io.ReadAll(resp.Body)
	if err != nil {
		return "", errors.Wrap(err, "failed to read image data")
	}

	// 检测图片格式
	kind, err := filetype.Match(imageData)
	if err != nil {
		return "", errors.Wrap(err, "failed to detect file type")
	}

	if !filetype.IsImage(imageData) {
		return "", errors.New("downloaded file is not a valid image")
	}

View on GitHub (pinned to 332d196854)

Solutions

  1. Log the actual status code and URL; for 403 check whether the site requires the UA/Referer headers this downloader already sends, and whether the signed URL expired
  2. For 429, add throttling/delay between requests and retry with exponential backoff
  3. Re-parse the page to get fresh image URLs if they are expiring or the image was moved (404)
  4. Verify the URL scheme is correct (http vs https, no relative path passed through unchanged)
  5. For persistent 5xx, retry later or skip the image and continue the batch (DownloadImages already collects per-URL errors)

Example fix

// before
if resp.StatusCode != http.StatusOK {
	return "", fmt.Errorf("download failed with status %d for URL: %s", resp.StatusCode, imageURL)
}
// after — retry on transient statuses with backoff
if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {
	time.Sleep(backoff)
	return d.DownloadImage(imageURL) // retry
}
if resp.StatusCode != http.StatusOK {
	return "", fmt.Errorf("download failed with status %d for URL: %s", resp.StatusCode, imageURL)
}
Defensive patterns

Strategy: retry

Validate before calling

resp, err := http.Head(imageURL)
if err != nil || resp.StatusCode != http.StatusOK {
	// 图片不可达（403/404/429），跳过或换源
}

Try / catch

path, err := d.DownloadImage(imageURL)
if err != nil {
	var statusErr interface{ Error() string }
	_ = statusErr
	if strings.Contains(err.Error(), "status 429") || strings.Contains(err.Error(), "status 5") {
		time.Sleep(2 * time.Second)
		path, err = d.DownloadImage(imageURL) // 退避重试一次
	}
	if err != nil { log.Printf("skip %s: %v", imageURL, err) }
}

Prevention

When it happens

Trigger: DownloadImages or ProcessImages calls d.DownloadImage(imageURL); the HTTP request succeeds but the server responds with e.g. 404 (image moved/deleted), 403 (hotlink protection / missing UA or Referer — note the UA/Referer headers this client sends), 429 (rate limited), or 5xx.

Common situations: Hotlink protection rejecting requests without proper Referer/User-Agent; images behind expiring signed URLs (S3/CDN token expired -> 403); rate limiting (429) when scraping many images from one host; image removed after the page HTML was parsed (stale URL); geo-blocked CDN.

Related errors

AI-assisted analysis of xpzouying/xiaohongshu-mcp@332d196854 (2026-09-05). Data as JSON: /api/errors/4de7e425895cede1. Report an issue: GitHub.