gocolly/colly · error

ErrForbiddenDomain

ErrForbiddenDomain

Error message

Forbidden domain

What it means

ErrForbiddenDomain is returned when the collector is asked to visit a URL whose host is not listed in the collector's AllowedDomains setting. Colly checks AllowedDomains (if non-empty) before issuing any HTTP request and refuses to crawl off-list hosts, as a scope/whitelist safety mechanism. The visit fails and the error surfaces via the OnError callback or the error returned by Visit().

Source

Thrown at colly.go:223

var collectorCounter uint32

// The key type is unexported to prevent collisions with context keys defined in
// other packages.
type key int

// ProxyURLKey is the context key for the request proxy address.
const (
	ProxyURLKey key = iota
	CheckRevisitKey
)

// The prefix for environment variables of Colly settings
const envVariablePrefix = "COLLY_"

var (
	// ErrForbiddenDomain is the error thrown if visiting
	// a domain which is not allowed in AllowedDomains
	ErrForbiddenDomain = errors.New("Forbidden domain")
	// ErrMissingURL is the error type for missing URL errors
	ErrMissingURL = errors.New("Missing URL")
	// ErrMaxDepth is the error type for exceeding max depth
	ErrMaxDepth = errors.New("Max depth limit reached")
	// ErrForbiddenURL is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrForbiddenURL = errors.New("ForbiddenURL")

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Add the exact hostname of the URL you are visiting to c.AllowedDomains
2. Remove or empty AllowedDomains entirely if you do not want domain restriction (no whitelist means all domains allowed)
3. Check for subdomain/scheme mismatches: entries match the literal Host, so list both example.com and www.example.com
4. Verify the URL you pass to Visit is the one you intend — redirects to off-list domains also trigger this

Example fix

// before
c := colly.NewCollector(colly.AllowedDomains("example.com"))
c.Visit("https://www.example.com/page") // Forbidden domain
// after
c := colly.NewCollector(colly.AllowedDomains("example.com", "www.example.com"))
c.Visit("https://www.example.com/page")

u, _ := url.Parse(target)
allowed := []string{"example.com", "www.example.com"}
if !slices.Contains(allowed, u.Hostname()) {
    return fmt.Errorf("skipping %s: domain not in AllowedDomains", u.Host)
}

func isAllowedDomain(u *url.URL, allowed []string) bool {
    return slices.Contains(allowed, u.Hostname())
}

err := c.Visit(target)
if err != nil && errors.Is(err, colly.ErrForbiddenDomain) {
    log.Printf("domain %s not whitelisted, skipping", target)
    return nil
}

When it happens

Trigger: Calling c.Visit() (or c.Request()) on a URL whose hostname is not in c.AllowedDomains; c.AllowedDomains is set and the target redirects or links to another domain that is not whitelisted; case/subdomain mismatch (www.example.com vs example.com) against an exact-match AllowDomains entry.

Common situations: Developers set AllowedDomains("example.com") then follow links to cdn.example.com or www.example.com and get blocked; scraping a site that redirects to a different domain (e.g. to https or a country TLD); forgetting to update the whitelist when the crawl target moves domains; calling a test harness (TestCollectorVisitWithAllowedDomains/DisallowedDomains) that exercises this check.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• ErrMissingURL
• ErrMaxDepth
• ErrForbiddenURL
• ErrNoURLFiltersMatch
• ErrRobotsTxtBlocked

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/e2fda0723a99098d. Report an issue: GitHub.