gocolly/colly · error

ErrMaxRequests

ErrMaxRequests

Error message

Max Requests limit reached

What it means

ErrMaxRequests is returned by the collector's requestCheck when MaxRequests is set (>0) and the collector has already issued MaxRequests requests (requestCount >= MaxRequests). It enforces a hard cap on total requests per collector and further requests are refused. Declared as a sentinel in colly.go:250.

Source

Thrown at colly.go:250

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list
	ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
	// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
	// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedBeforeRequest = errors.New("Aborted before Do Request")
	// ErrQueueFull is the error returned when the queue is full
	ErrQueueFull = errors.New("Queue MaxSize reached")
	// ErrMaxRequests is the error returned when exceeding max requests
	ErrMaxRequests = errors.New("Max Requests limit reached")
	// ErrRetryBodyUnseekable is the error when retry with not seekable body
	ErrRetryBodyUnseekable = errors.New("Retry Body Unseekable")
)

var envMap = map[string]func(*Collector, string){
	"ALLOWED_DOMAINS": func(c *Collector, val string) {
		c.AllowedDomains = strings.Split(val, ",")
	},
	"CACHE_DIR": func(c *Collector, val string) {
		c.CacheDir = val
	},
	"DETECT_CHARSET": func(c *Collector, val string) {
		c.DetectCharset = isYesString(val)
	},
	"DISABLE_COOKIES": func(c *Collector, _ string) {
		c.backend.Client.Jar = nil
	},
	"DISALLOWED_DOMAINS": func(c *Collector, val string) {

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Raise c.MaxRequests or set it to 0 (unlimited) if the cap is too low
2. Create a fresh Collector when you want a new request budget
3. Check whether MAX_REQUESTS env var is set in the environment
4. Treat the error as a normal stop condition: use errors.Is(err, colly.ErrMaxRequests) to end the crawl gracefully

Example fix

// before
c := colly.NewCollector(colly.MaxRequests(10))
// after
c := colly.NewCollector() // no cap, or a larger budget
c.MaxRequests = 1000

if c.MaxRequests > 0 && remaining > c.MaxRequests {
    // split work across collectors or raise the cap before starting
}

func isMaxRequests(err error) bool { return errors.Is(err, colly.ErrMaxRequests) }

if err := c.Visit(url); err != nil {
    if errors.Is(err, colly.ErrMaxRequests) {
        log.Println("request budget exhausted, stopping crawl")
        return
    }
    return err
}

When it happens

Trigger: Calling c.Visit/c.Do (anything passing requestCheck) after the collector already performed c.MaxRequests requests; also triggered via env var MAX_REQUESTS configuration.

Common situations: Long-running scrapers reused across many URLs where the cap is hit mid-crawl; setting MaxRequests via environment variable and forgetting it applies to the whole collector lifetime, not per crawl session.

Related errors

• ErrNoPattern
• ErrForbiddenDomain
• ErrMissingURL
• ErrMaxDepth
• ErrForbiddenURL

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/d54f92f8afcebb67. Report an issue: GitHub.