{"record":{"id":"8212218d7ed7761e","repo":"gocolly/colly","slug":"errqueuefull","errorCode":"ErrQueueFull","errorMessage":"Queue MaxSize reached","messagePattern":"Queue MaxSize reached","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"colly.go","lineNumber":248,"sourceCode":"\tErrForbiddenURL = errors.New(\"ForbiddenURL\")\n\n\t// ErrNoURLFiltersMatch is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrNoURLFiltersMatch = errors.New(\"No URLFilters match\")\n\t// ErrRobotsTxtBlocked is the error type for robots.txt errors\n\tErrRobotsTxtBlocked = errors.New(\"URL blocked by robots.txt\")\n\t// ErrNoCookieJar is the error type for missing cookie jar\n\tErrNoCookieJar = errors.New(\"Cookie jar is not available\")\n\t// ErrNoPattern is the error type for LimitRules without patterns\n\tErrNoPattern = errors.New(\"No pattern defined in LimitRule\")\n\t// ErrEmptyProxyURL is the error type for empty Proxy URL list\n\tErrEmptyProxyURL = errors.New(\"Proxy URL list is empty\")\n\t// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedAfterHeaders = errors.New(\"Aborted after receiving response headers\")\n\t// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedBeforeRequest = errors.New(\"Aborted before Do Request\")\n\t// ErrQueueFull is the error returned when the queue is full\n\tErrQueueFull = errors.New(\"Queue MaxSize reached\")\n\t// ErrMaxRequests is the error returned when exceeding max requests\n\tErrMaxRequests = errors.New(\"Max Requests limit reached\")\n\t// ErrRetryBodyUnseekable is the error when retry with not seekable body\n\tErrRetryBodyUnseekable = errors.New(\"Retry Body Unseekable\")\n)\n\nvar envMap = map[string]func(*Collector, string){\n\t\"ALLOWED_DOMAINS\": func(c *Collector, val string) {\n\t\tc.AllowedDomains = strings.Split(val, \",\")\n\t},\n\t\"CACHE_DIR\": func(c *Collector, val string) {\n\t\tc.CacheDir = val\n\t},\n\t\"DETECT_CHARSET\": func(c *Collector, val string) {\n\t\tc.DetectCharset = isYesString(val)\n\t},\n\t\"DISABLE_COOKIES\": func(c *Collector, _ string) {\n\t\tc.backend.Client.Jar = nil","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/gocolly/colly/blob/17d1d6ca92bd32a5651f34256bf7a2855c967f65/colly.go#L230-L266","documentation":"ErrQueueFull is returned by queue.Queue.AddRequest when the queue's MaxSize is greater than zero and the queue already holds MaxSize entries. The in-memory queue refuses to enqueue further requests so memory usage stays bounded. It is a back-pressure signal, not a network failure.","triggerScenarios":"Calling queue.AddRequest (directly or via storage) when q.size >= q.MaxSize, typically after a scraper enqueues more URLs than the configured queue capacity.","commonSituations":"Large crawls where the site yields more links than expected, MaxSize left at a small default or set too low, or a stalled consumer (limited workers, slow responses) causing the producer to outpace it.","solutions":["Increase the queue's MaxSize (e.g. q.MaxSize = 100000) to fit the expected crawl size","Set MaxSize to 0 (unlimited) if memory allows","Check why requests are not being consumed fast enough (worker count, LimitRules) and speed up draining","Handle the error in AddRequest call sites — retry enqueueing later or log-and-skip the URL"],"exampleFix":"// before\nq, _ := queue.New(100)\n// after\nq, _ := queue.New(0) // unlimited, or size to expected crawl\n","handlingStrategy":"validation","validationCode":"if q.MaxSize > 0 && q.Size() >= q.MaxSize {\n    // grow the queue, drain first, or skip enqueueing\n}","typeGuard":null,"tryCatchPattern":"if err := q.AddRequest(r); err != nil {\n    if errors.Is(err, colly.ErrQueueFull) {\n        log.Println(\"queue full, dropping/skipping\")\n        return\n    }\n    return err\n}","preventionTips":["Size MaxSize to the expected number of discovered URLs before a crawl","Monitor queue size and drain rate; add workers if producers outpace consumers","Set MaxSize 0 when unbounded memory is acceptable","Treat AddRequest's error as required handling, never ignore it"],"tags":["queue","backpressure","colly"],"backgroundTag":"queue-full","analyzedSha":"17d1d6ca92bd32a5651f34256bf7a2855c967f65","analyzedAt":"2026-08-30T21:32:31.579Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}