fish2018/pansou · error

429 Too Many Requests

Error message

429 Too Many Requests

What it means

doRequestWithRateLimitRetry retries a JAVDB HTTP request when the server answers HTTP 429 (rate limited). On the final retry attempt, instead of returning the 429-specific wrapped error, the loop sets lastErr = "429 Too Many Requests" and falls through to `return nil, lastErr` — so the caller receives a bare 429 error after exhausting MaxRetryOnRateLimit retries with random backoff delays.

Solutions

1. Increase delays between JAVDB requests or add global rate limiting across goroutines
2. Reduce concurrent search/detail workers or add a shared cooldown when p.rateLimited is set
3. Honor the existing random backoff (MinRetryDelay/MaxRetryDelay) and raise MaxRetryOnRateLimit only if the site tolerates it
4. Check for proxy/ban status — a persistent 429 may be IP-level throttling; rotate proxies or wait longer
5. Propagate the richer 429-limited error (line 315 branch) by bounding attempts correctly instead of leaking the bare lastErr

Example fix

// before
lastErr = fmt.Errorf("429 Too Many Requests")
}
return nil, lastErr

// after
lastErr = fmt.Errorf("[%s] 429 Too Many Requests after %d retries", p.Name(), MaxRetryOnRateLimit)
}
return nil, lastErr

// Go: caller of fetchDetailPageMagnetLinks
results, err := p.fetchDetailPageMagnetLinks(url, client)
if err != nil {
	if strings.Contains(err.Error(), "429") {
		select {
		case <-time.After(cooldown):
		case <-ctx.Done():
			return ctx.Err()
		}
		results, err = p.fetchDetailPageMagnetLinks(url, client)
	}
	if err != nil {
		return nil, err
	}
}

When it happens

Trigger: fetchDetailPageMagnetLinks requests a JAVDB detail page and every attempt (initial + MaxRetryOnRateLimit retries) receives HTTP 429 from the JAVDB server, exhausting the retry loop.

Common situations: Scraping too many detail pages too fast; many plugin goroutines querying JAVDB concurrently from one IP; JAVDB tightening anti-bot rate limits; long-running crawls without cooldown triggering IP-based throttling.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• [ ] 第 页请求返回状态码
• status
• HTTP 状态码
• HTTP 状态码
• [ ] 搜索请求失败

AI-assisted analysis of fish2018/pansou@beaa561337 (2026-09-07). Data as JSON: /api/errors/77c110843b267106. Report an issue: GitHub.

Appendix: source

Thrown at plugin/javdb/javdb.go:325

		if p.debugMode {
			log.Printf("[JAVDB] 遇到429限流，尝试 %d/%d", attempt+1, MaxRetryOnRateLimit+1)
		}
		
		// 如果不允许重试或已达到最大重试次数
		if MaxRetryOnRateLimit == 0 || attempt >= MaxRetryOnRateLimit {
			atomic.StoreInt32(&p.rateLimited, 1)
			resp.Body.Close()
			return nil, fmt.Errorf("[%s] 429限流，%s", p.Name(), 
				func() string {
					if MaxRetryOnRateLimit == 0 {
						return "不重试"
					}
					return fmt.Sprintf("重试%d次后仍然限流", MaxRetryOnRateLimit)
				}())
		}
		
		resp.Body.Close()
		lastErr = fmt.Errorf("429 Too Many Requests")
	}
	
	return nil, lastErr
}

// parseSearchResults 解析搜索结果HTML
func (p *JavdbPlugin) parseSearchResults(doc *goquery.Document) ([]model.SearchResult, error) {
	var results []model.SearchResult

	if p.debugMode {
		// 检查是否找到了.movie-list元素
		movieListEl := doc.Find(".movie-list")
		log.Printf("[JAVDB] 找到.movie-list元素数量: %d", movieListEl.Length())
		
		// 检查是否找到了.item元素
		itemEls := doc.Find(".movie-list .item")
		log.Printf("[JAVDB] 找到.movie-list .item元素数量: %d", itemEls.Length())
		

View on GitHub (pinned to beaa561337)