{"record":{"id":"bec89687f3fc6b1b","repo":"gocolly/colly","slug":"errforbiddenurl","errorCode":"ErrForbiddenURL","errorMessage":"ForbiddenURL","messagePattern":"ForbiddenURL","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"colly.go","lineNumber":230,"sourceCode":"const (\n\tProxyURLKey key = iota\n\tCheckRevisitKey\n)\n\n// The prefix for environment variables of Colly settings\nconst envVariablePrefix = \"COLLY_\"\n\nvar (\n\t// ErrForbiddenDomain is the error thrown if visiting\n\t// a domain which is not allowed in AllowedDomains\n\tErrForbiddenDomain = errors.New(\"Forbidden domain\")\n\t// ErrMissingURL is the error type for missing URL errors\n\tErrMissingURL = errors.New(\"Missing URL\")\n\t// ErrMaxDepth is the error type for exceeding max depth\n\tErrMaxDepth = errors.New(\"Max depth limit reached\")\n\t// ErrForbiddenURL is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrForbiddenURL = errors.New(\"ForbiddenURL\")\n\n\t// ErrNoURLFiltersMatch is the error thrown if visiting\n\t// a URL which is not allowed by URLFilters\n\tErrNoURLFiltersMatch = errors.New(\"No URLFilters match\")\n\t// ErrRobotsTxtBlocked is the error type for robots.txt errors\n\tErrRobotsTxtBlocked = errors.New(\"URL blocked by robots.txt\")\n\t// ErrNoCookieJar is the error type for missing cookie jar\n\tErrNoCookieJar = errors.New(\"Cookie jar is not available\")\n\t// ErrNoPattern is the error type for LimitRules without patterns\n\tErrNoPattern = errors.New(\"No pattern defined in LimitRule\")\n\t// ErrEmptyProxyURL is the error type for empty Proxy URL list\n\tErrEmptyProxyURL = errors.New(\"Proxy URL list is empty\")\n\t// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedAfterHeaders = errors.New(\"Aborted after receiving response headers\")\n\t// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.\n\tErrAbortedBeforeRequest = errors.New(\"Aborted before Do Request\")\n\t// ErrQueueFull is the error returned when the queue is full\n\tErrQueueFull = errors.New(\"Queue MaxSize reached\")","sourceCodeStart":212,"sourceCodeEnd":248,"githubUrl":"https://github.com/gocolly/colly/blob/17d1d6ca92bd32a5651f34256bf7a2855c967f65/colly.go#L212-L248","documentation":"ErrForbiddenURL is returned when the URL to visit is rejected by the collector's URLFilters (URLFilter/DisallowURLFilters rules). Colly evaluates the filter rules against the request URL before sending it and blocks any URL that is not allowed. Like ErrForbiddenDomain it is a scope guard, applied at URL level (including path/query) rather than at host level.","triggerScenarios":"Calling c.Visit() on a URL excluded by c.URLFilter() rules; a URL matched by c.DisallowURLFilters(); following a link in OnHTML that fails the configured URL patterns; regex/path rules that accidentally match more URLs than expected.","commonSituations":"Filtering out file extensions (.jpg, .pdf) but a link unexpectedly matches; using regex filters whose pattern is too broad or too narrow; crawlers that inherit a previous collector's URLFilter rules and then try to visit new URL shapes.","solutions":["Inspect the URL against your URLFilter/DisallowURLFilters patterns and adjust the rule to allow it","Add an allow pattern with c.URLFilter for the URL form you actually want","Remove or narrow DisallowURLFilters entries that over-match","Log u in OnRequest (or print the rejected URL) to see exactly which shape is being blocked"],"exampleFix":"// before\nc := colly.NewCollector()\nc.URLFilter = func(u *url.URL) bool { return u.Path == \"/articles\" } // too strict\nc.Visit(\"https://example.com/articles/123\") // ForbiddenURL\n// after\nc := colly.NewCollector()\nc.URLFilter = func(u *url.URL) bool { return strings.HasPrefix(u.Path, \"/articles\") }\nc.Visit(\"https://example.com/articles/123\")","handlingStrategy":"validation","validationCode":"u, _ := url.Parse(target)\nif c.URLFilter != nil && !c.URLFilter(u) {\n    return fmt.Errorf(\"skipping %s: blocked by URLFilter\", target)\n}","typeGuard":null,"tryCatchPattern":"if err := c.Visit(target); err != nil && errors.Is(err, colly.ErrForbiddenURL) {\n    log.Printf(\"URL filtered out: %s\", target)\n    return nil\n}","preventionTips":["Unit-test URLFilter/DisallowURLFilters patterns against real URL samples","Log filtered URLs in OnRequest to spot over-broad rules","Keep filter rules with the crawl config and review regex scope"],"tags":["colly","scraping","url-filter","go"],"backgroundTag":"url-blocked-by-filter","analyzedSha":"17d1d6ca92bd32a5651f34256bf7a2855c967f65","analyzedAt":"2026-08-30T21:32:31.579Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}