{"record":{"id":"e2fda0723a99098d","repo":"gocolly/colly","slug":"errforbiddendomain","errorCode":"ErrForbiddenDomain","errorMessage":"Forbidden domain","messagePattern":"Forbidden domain","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"colly.go","lineNumber":223,"sourceCode":"var collectorCounter uint32\n\n// The key type is unexported to prevent collisions with context keys defined in\n// other packages.\ntype key int\n\n// ProxyURLKey is the context key for the request proxy address.\nconst (\n\tProxyURLKey key = iota\n\tCheckRevisitKey\n)\n\n// The prefix for environment variables of Colly settings\nconst envVariablePrefix = \"COLLY_\"\n\nvar (\n\t// ErrForbiddenDomain is the error thrown if visiting\n\t// a domain which is not allowed in AllowedDomains\n\tErrForbiddenDomain = errors.New(\"Forbidden domain\")\n\t// ErrMissingURL is the error type for missing URL errors\n\tErrMissingURL = errors.New(\"Missing URL\")\n\t// ErrMaxDepth is the error type for exceeding max depth\n\tErrMaxDepth = errors.New(\"Max depth limit reached\")\n\t// ErrForbiddenURL is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrForbiddenURL = errors.New(\"ForbiddenURL\")\n\n\t// ErrNoURLFiltersMatch is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrNoURLFiltersMatch = errors.New(\"No URLFilters match\")\n\t// ErrRobotsTxtBlocked is the error type for robots.txt errors\n\tErrRobotsTxtBlocked = errors.New(\"URL blocked by robots.txt\")\n\t// ErrNoCookieJar is the error type for missing cookie jar\n\tErrNoCookieJar = errors.New(\"Cookie jar is not available\")\n\t// ErrNoPattern is the error type for LimitRules without patterns\n\tErrNoPattern = errors.New(\"No pattern defined in LimitRule\")\n\t// ErrEmptyProxyURL is the error type for empty Proxy URL list","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/gocolly/colly/blob/17d1d6ca92bd32a5651f34256bf7a2855c967f65/colly.go#L205-L241","documentation":"ErrForbiddenDomain is returned when the collector is asked to visit a URL whose host is not listed in the collector's AllowedDomains setting. Colly checks AllowedDomains (if non-empty) before issuing any HTTP request and refuses to crawl off-list hosts, as a scope/whitelist safety mechanism. The visit fails and the error surfaces via the OnError callback or the error returned by Visit().","triggerScenarios":"Calling c.Visit() (or c.Request()) on a URL whose hostname is not in c.AllowedDomains; c.AllowedDomains is set and the target redirects or links to another domain that is not whitelisted; case/subdomain mismatch (www.example.com vs example.com) against an exact-match AllowDomains entry.","commonSituations":"Developers set AllowedDomains(\"example.com\") then follow links to cdn.example.com or www.example.com and get blocked; scraping a site that redirects to a different domain (e.g. to https or a country TLD); forgetting to update the whitelist when the crawl target moves domains; calling a test harness (TestCollectorVisitWithAllowedDomains/DisallowedDomains) that exercises this check.","solutions":["Add the exact hostname of the URL you are visiting to c.AllowedDomains","Remove or empty AllowedDomains entirely if you do not want domain restriction (no whitelist means all domains allowed)","Check for subdomain/scheme mismatches: entries match the literal Host, so list both example.com and www.example.com","Verify the URL you pass to Visit is the one you intend — redirects to off-list domains also trigger this"],"exampleFix":"// before\nc := colly.NewCollector(colly.AllowedDomains(\"example.com\"))\nc.Visit(\"https://www.example.com/page\") // Forbidden domain\n// after\nc := colly.NewCollector(colly.AllowedDomains(\"example.com\", \"www.example.com\"))\nc.Visit(\"https://www.example.com/page\")","handlingStrategy":"validation","validationCode":"u, _ := url.Parse(target)\nallowed := []string{\"example.com\", \"www.example.com\"}\nif !slices.Contains(allowed, u.Hostname()) {\n    return fmt.Errorf(\"skipping %s: domain not in AllowedDomains\", u.Host)\n}","typeGuard":"func isAllowedDomain(u *url.URL, allowed []string) bool {\n    return slices.Contains(allowed, u.Hostname())\n}","tryCatchPattern":"err := c.Visit(target)\nif err != nil && errors.Is(err, colly.ErrForbiddenDomain) {\n    log.Printf(\"domain %s not whitelisted, skipping\", target)\n    return nil\n}","preventionTips":["List every hostname variant (apex + www + CDNs) in AllowedDomains","Test redirects of your target URLs against the whitelist","Keep AllowedDomains in one config place and review it on crawl-target changes"],"tags":["colly","scraping","domain-whitelist","go"],"backgroundTag":"http-403-forbidden-domain","analyzedSha":"17d1d6ca92bd32a5651f34256bf7a2855c967f65","analyzedAt":"2026-08-30T21:32:31.579Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}