gocolly/colly · error

ErrForbiddenURL

ErrForbiddenURL

Error message

ForbiddenURL

What it means

ErrForbiddenURL is returned when the URL to visit is rejected by the collector's URLFilters (URLFilter/DisallowURLFilters rules). Colly evaluates the filter rules against the request URL before sending it and blocks any URL that is not allowed. Like ErrForbiddenDomain it is a scope guard, applied at URL level (including path/query) rather than at host level.

Source

Thrown at colly.go:230

const (
	ProxyURLKey key = iota
	CheckRevisitKey
)

// The prefix for environment variables of Colly settings
const envVariablePrefix = "COLLY_"

var (
	// ErrForbiddenDomain is the error thrown if visiting
	// a domain which is not allowed in AllowedDomains
	ErrForbiddenDomain = errors.New("Forbidden domain")
	// ErrMissingURL is the error type for missing URL errors
	ErrMissingURL = errors.New("Missing URL")
	// ErrMaxDepth is the error type for exceeding max depth
	ErrMaxDepth = errors.New("Max depth limit reached")
	// ErrForbiddenURL is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrForbiddenURL = errors.New("ForbiddenURL")

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list
	ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
	// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
	// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedBeforeRequest = errors.New("Aborted before Do Request")
	// ErrQueueFull is the error returned when the queue is full
	ErrQueueFull = errors.New("Queue MaxSize reached")

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Inspect the URL against your URLFilter/DisallowURLFilters patterns and adjust the rule to allow it
2. Add an allow pattern with c.URLFilter for the URL form you actually want
3. Remove or narrow DisallowURLFilters entries that over-match
4. Log u in OnRequest (or print the rejected URL) to see exactly which shape is being blocked

Example fix

// before
c := colly.NewCollector()
c.URLFilter = func(u *url.URL) bool { return u.Path == "/articles" } // too strict
c.Visit("https://example.com/articles/123") // ForbiddenURL
// after
c := colly.NewCollector()
c.URLFilter = func(u *url.URL) bool { return strings.HasPrefix(u.Path, "/articles") }
c.Visit("https://example.com/articles/123")

u, _ := url.Parse(target)
if c.URLFilter != nil && !c.URLFilter(u) {
    return fmt.Errorf("skipping %s: blocked by URLFilter", target)
}

if err := c.Visit(target); err != nil && errors.Is(err, colly.ErrForbiddenURL) {
    log.Printf("URL filtered out: %s", target)
    return nil
}

When it happens

Trigger: Calling c.Visit() on a URL excluded by c.URLFilter() rules; a URL matched by c.DisallowURLFilters(); following a link in OnHTML that fails the configured URL patterns; regex/path rules that accidentally match more URLs than expected.

Common situations: Filtering out file extensions (.jpg, .pdf) but a link unexpectedly matches; using regex filters whose pattern is too broad or too narrow; crawlers that inherit a previous collector's URLFilter rules and then try to visit new URL shapes.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• ErrNoURLFiltersMatch
• ErrForbiddenDomain
• ErrMissingURL
• ErrMaxDepth
• ErrRobotsTxtBlocked

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/bec89687f3fc6b1b. Report an issue: GitHub.