{"record":{"id":"d54f92f8afcebb67","repo":"gocolly/colly","slug":"errmaxrequests","errorCode":"ErrMaxRequests","errorMessage":"Max Requests limit reached","messagePattern":"Max Requests limit reached","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"colly.go","lineNumber":250,"sourceCode":"\t// ErrNoURLFiltersMatch is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrNoURLFiltersMatch = errors.New(\"No URLFilters match\")\n\t// ErrRobotsTxtBlocked is the error type for robots.txt errors\n\tErrRobotsTxtBlocked = errors.New(\"URL blocked by robots.txt\")\n\t// ErrNoCookieJar is the error type for missing cookie jar\n\tErrNoCookieJar = errors.New(\"Cookie jar is not available\")\n\t// ErrNoPattern is the error type for LimitRules without patterns\n\tErrNoPattern = errors.New(\"No pattern defined in LimitRule\")\n\t// ErrEmptyProxyURL is the error type for empty Proxy URL list\n\tErrEmptyProxyURL = errors.New(\"Proxy URL list is empty\")\n\t// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedAfterHeaders = errors.New(\"Aborted after receiving response headers\")\n\t// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedBeforeRequest = errors.New(\"Aborted before Do Request\")\n\t// ErrQueueFull is the error returned when the queue is full\n\tErrQueueFull = errors.New(\"Queue MaxSize reached\")\n\t// ErrMaxRequests is the error returned when exceeding max requests\n\tErrMaxRequests = errors.New(\"Max Requests limit reached\")\n\t// ErrRetryBodyUnseekable is the error when retry with not seekable body\n\tErrRetryBodyUnseekable = errors.New(\"Retry Body Unseekable\")\n)\n\nvar envMap = map[string]func(*Collector, string){\n\t\"ALLOWED_DOMAINS\": func(c *Collector, val string) {\n\t\tc.AllowedDomains = strings.Split(val, \",\")\n\t},\n\t\"CACHE_DIR\": func(c *Collector, val string) {\n\t\tc.CacheDir = val\n\t},\n\t\"DETECT_CHARSET\": func(c *Collector, val string) {\n\t\tc.DetectCharset = isYesString(val)\n\t},\n\t\"DISABLE_COOKIES\": func(c *Collector, _ string) {\n\t\tc.backend.Client.Jar = nil\n\t},\n\t\"DISALLOWED_DOMAINS\": func(c *Collector, val string) {","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/gocolly/colly/blob/17d1d6ca92bd32a5651f34256bf7a2855c967f65/colly.go#L232-L268","documentation":"ErrMaxRequests is returned by the collector's requestCheck when MaxRequests is set (>0) and the collector has already issued MaxRequests requests (requestCount >= MaxRequests). It enforces a hard cap on total requests per collector and further requests are refused. Declared as a sentinel in colly.go:250.","triggerScenarios":"Calling c.Visit/c.Do (anything passing requestCheck) after the collector already performed c.MaxRequests requests; also triggered via env var MAX_REQUESTS configuration.","commonSituations":"Long-running scrapers reused across many URLs where the cap is hit mid-crawl; setting MaxRequests via environment variable and forgetting it applies to the whole collector lifetime, not per crawl session.","solutions":["Raise c.MaxRequests or set it to 0 (unlimited) if the cap is too low","Create a fresh Collector when you want a new request budget","Check whether MAX_REQUESTS env var is set in the environment","Treat the error as a normal stop condition: use errors.Is(err, colly.ErrMaxRequests) to end the crawl gracefully"],"exampleFix":"// before\nc := colly.NewCollector(colly.MaxRequests(10))\n// after\nc := colly.NewCollector() // no cap, or a larger budget\nc.MaxRequests = 1000","handlingStrategy":"try-catch","validationCode":"if c.MaxRequests > 0 && remaining > c.MaxRequests {\n    // split work across collectors or raise the cap before starting\n}","typeGuard":"func isMaxRequests(err error) bool { return errors.Is(err, colly.ErrMaxRequests) }","tryCatchPattern":"if err := c.Visit(url); err != nil {\n    if errors.Is(err, colly.ErrMaxRequests) {\n        log.Println(\"request budget exhausted, stopping crawl\")\n        return\n    }\n    return err\n}","preventionTips":["Budget MaxRequests against the size of your crawl plan","Check the MAX_REQUESTS environment variable in deployment configs","Create a new Collector per crawl session if the cap is meant to be per-run","Handle the sentinel as a graceful stop condition"],"tags":["rate-limit","request-budget","colly"],"backgroundTag":"request-limit-reached","analyzedSha":"17d1d6ca92bd32a5651f34256bf7a2855c967f65","analyzedAt":"2026-08-30T21:32:31.579Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}