What causes "Max depth limit reached" in gocolly/colly?

Increase the limit: c.MaxDepth(n) with n large enough for your crawl tree

ErrMaxDepth

ErrMaxDepth

Error message

Max depth limit reached

What it means

ErrMaxDepth is returned when a link passed to Visit() would exceed the collector's MaxDepth limit. Colly tracks the crawl depth of each request (depth 0 for direct Visit calls, incremented per followed link via OnHTML->Visit) and refuses to crawl deeper than the configured limit. This is an intentional boundary, not a network failure.

Source

Thrown at colly.go:227

type key int

// ProxyURLKey is the context key for the request proxy address.
const (
	ProxyURLKey key = iota
	CheckRevisitKey
)

// The prefix for environment variables of Colly settings
const envVariablePrefix = "COLLY_"

var (
	// ErrForbiddenDomain is the error thrown if visiting
	// a domain which is not allowed in AllowedDomains
	ErrForbiddenDomain = errors.New("Forbidden domain")
	// ErrMissingURL is the error type for missing URL errors
	ErrMissingURL = errors.New("Missing URL")
	// ErrMaxDepth is the error type for exceeding max depth
	ErrMaxDepth = errors.New("Max depth limit reached")
	// ErrForbiddenURL is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrForbiddenURL = errors.New("ForbiddenURL")

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list
	ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
	// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
	// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.

View on GitHub (pinned to 17d1d6ca92)

Solutions

Increase the limit: c.MaxDepth(n) with n large enough for your crawl tree
Guard recursion: inside the handler check c.Depth() < maxDepth before calling Visit
If you only need depth-limited traversal, keep MaxDepth but accept the error as the stop signal
If unlimited crawling is intended, call c.MaxDepth(0)... note depth checks only apply when MaxDepth is set >0 — otherwise remove manual recursive Visit calls

Example fix

// before
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    c.Visit(e.Request.AbsoluteURL(e.Attr("href"))) // ErrMaxDepth past limit
})
// after
c.MaxDepth(3)
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    if c.Depth() < 3 {
        c.Visit(e.Request.AbsoluteURL(e.Attr("href")))
    }
})

Defensive patterns

Strategy: validation

Validate before calling

const maxDepth = 3
if c.Depth() >= maxDepth {
    return nil // would trigger ErrMaxDepth
}

Try / catch

if err := c.Visit(u); err != nil && errors.Is(err, colly.ErrMaxDepth) {
    return nil // depth boundary reached, stop silently
}

Prevention

Set MaxDepth explicitly and compare c.Depth() before recursive Visits
Model your crawl tree depth before choosing MaxDepth
Prefer explicit depth checks over relying on the error as flow control

When it happens

Trigger: Calling c.Visit() from inside OnHTML/OnRequest handlers when c.Depth() already equals MaxDepth; setting c.MaxDepth(1) (or similar) and recursively following links; unbounded link-following loops that quickly hit the configured depth cap.

Common situations: Crawlers that only want the target page plus one level of links but whose handler keeps visiting; forgetting that depth counts from the initial Visit; recursive sitemap/category crawls whose structure is deeper than the configured MaxDepth; test TestCollectorMaxDepth-style exercises.

Related errors

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/df8756720aa3cf35. Report an issue: GitHub.