What causes "Queue MaxSize reached" in gocolly/colly?

Increase the queue's MaxSize (e.g. q.MaxSize = 100000) to fit the expected crawl size

ErrQueueFull

ErrQueueFull

Error message

Queue MaxSize reached

What it means

ErrQueueFull is returned by queue.Queue.AddRequest when the queue's MaxSize is greater than zero and the queue already holds MaxSize entries. The in-memory queue refuses to enqueue further requests so memory usage stays bounded. It is a back-pressure signal, not a network failure.

Source

Thrown at colly.go:248

	ErrForbiddenURL = errors.New("ForbiddenURL")

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list
	ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
	// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
	// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedBeforeRequest = errors.New("Aborted before Do Request")
	// ErrQueueFull is the error returned when the queue is full
	ErrQueueFull = errors.New("Queue MaxSize reached")
	// ErrMaxRequests is the error returned when exceeding max requests
	ErrMaxRequests = errors.New("Max Requests limit reached")
	// ErrRetryBodyUnseekable is the error when retry with not seekable body
	ErrRetryBodyUnseekable = errors.New("Retry Body Unseekable")
)

var envMap = map[string]func(*Collector, string){
	"ALLOWED_DOMAINS": func(c *Collector, val string) {
		c.AllowedDomains = strings.Split(val, ",")
	},
	"CACHE_DIR": func(c *Collector, val string) {
		c.CacheDir = val
	},
	"DETECT_CHARSET": func(c *Collector, val string) {
		c.DetectCharset = isYesString(val)
	},
	"DISABLE_COOKIES": func(c *Collector, _ string) {
		c.backend.Client.Jar = nil

View on GitHub (pinned to 17d1d6ca92)

Solutions

Increase the queue's MaxSize (e.g. q.MaxSize = 100000) to fit the expected crawl size
Set MaxSize to 0 (unlimited) if memory allows
Check why requests are not being consumed fast enough (worker count, LimitRules) and speed up draining
Handle the error in AddRequest call sites — retry enqueueing later or log-and-skip the URL

Example fix

// before
q, _ := queue.New(100)
// after
q, _ := queue.New(0) // unlimited, or size to expected crawl

Defensive patterns

Strategy: validation

Validate before calling

if q.MaxSize > 0 && q.Size() >= q.MaxSize {
    // grow the queue, drain first, or skip enqueueing
}

Try / catch

if err := q.AddRequest(r); err != nil {
    if errors.Is(err, colly.ErrQueueFull) {
        log.Println("queue full, dropping/skipping")
        return
    }
    return err
}

Prevention

Size MaxSize to the expected number of discovered URLs before a crawl
Monitor queue size and drain rate; add workers if producers outpace consumers
Set MaxSize 0 when unbounded memory is acceptable
Treat AddRequest's error as required handling, never ignore it

When it happens

Trigger: Calling queue.AddRequest (directly or via storage) when q.size >= q.MaxSize, typically after a scraper enqueues more URLs than the configured queue capacity.

Common situations: Large crawls where the site yields more links than expected, MaxSize left at a small default or set too low, or a stalled consumer (limited workers, slow responses) causing the producer to outpace it.

Related errors

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/8212218d7ed7761e. Report an issue: GitHub.