gocolly/colly · warning

ErrRobotsTxtBlocked

ErrRobotsTxtBlocked

Error message

URL blocked by robots.txt

What it means

ErrRobotsTxtBlocked is returned when the collector's CheckRobotsTxt option is enabled and the target site's robots.txt disallows crawling the requested URL. Colly fetches and honors robots.txt as a politeness/compliance mechanism before making the actual request. The crawl is refused locally — no page request is sent for the disallowed path.

Source

Thrown at colly.go:236

const envVariablePrefix = "COLLY_"

var (
	// ErrForbiddenDomain is the error thrown if visiting
	// a domain which is not allowed in AllowedDomains
	ErrForbiddenDomain = errors.New("Forbidden domain")
	// ErrMissingURL is the error type for missing URL errors
	ErrMissingURL = errors.New("Missing URL")
	// ErrMaxDepth is the error type for exceeding max depth
	ErrMaxDepth = errors.New("Max depth limit reached")
	// ErrForbiddenURL is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrForbiddenURL = errors.New("ForbiddenURL")

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list
	ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
	// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
	// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedBeforeRequest = errors.New("Aborted before Do Request")
	// ErrQueueFull is the error returned when the queue is full
	ErrQueueFull = errors.New("Queue MaxSize reached")
	// ErrMaxRequests is the error returned when exceeding max requests
	ErrMaxRequests = errors.New("Max Requests limit reached")
	// ErrRetryBodyUnseekable is the error when retry with not seekable body
	ErrRetryBodyUnseekable = errors.New("Retry Body Unseekable")
)

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Read the site's robots.txt and restrict your crawl to allowed paths
2. Remove colly.CheckRobotsTxt() only if you have permission to ignore robots.txt — prefer respecting it
3. Cache/handle the error in OnError and skip disallowed URL patterns instead of retrying
4. For allowed paths that still fail, verify the User-Agent matching the robots.txt rules (colly sends its default UA)

Example fix

// before
c := colly.NewCollector(colly.CheckRobotsTxt())
c.Visit("https://example.com/admin") // Disallow: /admin
// after
c := colly.NewCollector(colly.CheckRobotsTxt())
c.Visit("https://example.com/public/page") // allowed by robots.txt

// Pre-check robots.txt yourself before crawling
resp, err := http.Get(base + "/robots.txt")
// parse and confirm target path is not Disallowed for your UA

if err := c.Visit(u); err != nil && errors.Is(err, colly.ErrRobotsTxtBlocked) {
    log.Printf("robots.txt blocks %s, skipping", u)
    return nil
}

When it happens

Trigger: Creating a collector with colly.CheckRobotsTxt() and calling Visit() on a path disallowed by the site's robots.txt (Disallow rules or full-site disallow); visiting a URL under a Disallow prefix discovered mid-crawl via OnHTML link following; robots.txt updated server-side so previously crawlable URLs become blocked.

Common situations: Scraping endpoints like /admin, /search, or /api that sites commonly disallow; crawling during a site's heavy-traffic period when a blanket Disallow is active; enabling CheckRobotsTxt on an existing collector without auditing which target paths are blocked.

Related errors

• ErrForbiddenDomain
• ErrMissingURL
• ErrMaxDepth
• ErrForbiddenURL
• ErrNoURLFiltersMatch

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/cc8d3858c1d58499. Report an issue: GitHub.