{"record":{"id":"c5d021e473da0940","repo":"gocolly/colly","slug":"not-following-redirect-to-q-w","errorCode":null,"errorMessage":"Not following redirect to %q: %w","messagePattern":"Not following redirect to %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"colly.go","lineNumber":1464,"sourceCode":"\t\tdebugger:               c.debugger,\n\t\tAsync:                  c.Async,\n\t\tredirectHandler:        c.redirectHandler,\n\t\terrorCallbacks:         make([]ErrorCallback, 0, 8),\n\t\thtmlCallbacks:          make([]*htmlCallbackContainer, 0, 8),\n\t\txmlCallbacks:           make([]*xmlCallbackContainer, 0, 8),\n\t\tscrapedCallbacks:       make([]ScrapedCallback, 0, 8),\n\t\tlock:                   c.lock,\n\t\trequestCallbacks:       make([]RequestCallback, 0, 8),\n\t\tresponseCallbacks:      make([]ResponseCallback, 0, 8),\n\t\trobotsMap:              c.robotsMap,\n\t\twg:                     &sync.WaitGroup{},\n\t}\n}\n\nfunc (c *Collector) checkRedirectFunc() func(req *http.Request, via []*http.Request) error {\n\treturn func(req *http.Request, via []*http.Request) error {\n\t\tif err := c.checkFilters(req.URL.String(), req.URL.Hostname()); err != nil {\n\t\t\treturn fmt.Errorf(\"Not following redirect to %q: %w\", req.URL, err)\n\t\t}\n\n\t\t// Page may set cookies and respond with a redirect to itself.\n\t\t// Some example of such redirect \"cycles\":\n\t\t//\n\t\t// example.com -(set cookie)-> example.com\n\t\t// example.com -> auth.example.com -(set cookie)-> example.com\n\t\t// www.example.com -> example.com -(set cookie)-> example.com\n\t\t//\n\t\t// We must not return \"already visited\" error in such cases.\n\t\t// So ignore redirect cycles when checking for URL revisit.\n\t\tredirectCycle := false\n\t\tnormalizedURL := normalizeURL(req.URL.String())\n\t\tfor _, viaReq := range via {\n\t\t\tviaURL := normalizeURL(viaReq.URL.String())\n\t\t\tif viaURL == normalizedURL {\n\t\t\t\tredirectCycle = true\n\t\t\t\tbreak","sourceCodeStart":1446,"sourceCodeEnd":1482,"githubUrl":"https://github.com/gocolly/colly/blob/17d1d6ca92bd32a5651f34256bf7a2855c967f65/colly.go#L1446-L1482","documentation":"colly wraps any error returned by its redirect-check filter chain with \"Not following redirect to %q: %w\" inside the Collector's http.Client CheckRedirect func. It means colly deliberately refuses to follow an HTTP redirect because the redirect target (URL or host) fails URLFilters/DomainFilters or other allowed-domain checks. The wrapped inner error carries the actual reason (e.g. 'Forbidden domain').","triggerScenarios":"A request receives a 3xx redirect and the redirect target URL/host fails c.checkFilters: the target domain is not in AllowedDomains / not allowed by URLFilters, or the redirect chain exceeds depth limits. Happens with c.Visit, forms, or any request when the server redirects off-domain.","commonSituations":"Scraping sites that redirect to a login/SSO or CDN host not listed in AllowedDomains; http->https or www/non-www redirects to an unallowed variant; sites redirecting to an error page on another domain; configuring AllowedDomains too narrowly.","solutions":["Add the redirect target domain (and variants like www./CDN hosts) to Collector.AllowedDomains","Relax URLFilters that exclude the redirect target URL","Inspect the wrapped inner error (%w) to confirm which filter rejected the URL","If you intend to follow all redirects, clear over-restrictive domain filters"],"exampleFix":"// before\nc := colly.NewCollector(colly.AllowedDomains(\"example.com\"))\n// after (site redirects via www. and its CDN)\nc := colly.NewCollector(colly.AllowedDomains(\"example.com\", \"www.example.com\", \"cdn.example.com\"))","handlingStrategy":"try-catch","validationCode":"// before visiting, check the URL is allowed\nc := colly.NewCollector(colly.AllowedDomains(\"example.com\", \"www.example.com\"))\nif len(c.AllowedDomains) > 0 && !containsDomain(c.AllowedDomains, targetHost()) {\n    // target would be rejected on redirect\n}","typeGuard":"var redirectErr *colly.RedirectedError // if surfaced via c.OnError\nfunc isRedirectFilterErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"Not following redirect to\")\n}","tryCatchPattern":"c.OnError(func(r *colly.Response, err error) {\n    if strings.Contains(err.Error(), \"Not following redirect to\") {\n        log.Printf(\"redirect blocked to %s: %v\", r.Request.URL, err) // inspect wrapped cause\n        return\n    }\n    panic(err)\n})","preventionTips":["Enumerate every host the target site redirects to (www, https, CDN, SSO) and add all to AllowedDomains","Test the entry URL with curl -IL to observe the full redirect chain before scraping","Inspect the wrapped error (%w cause) to identify which filter rejected the URL","Avoid overly strict URLFilters that catch legitimate redirect targets"],"tags":["http","redirect","domain-filter","scraping"],"backgroundTag":"redirect-not-followed","analyzedSha":"17d1d6ca92bd32a5651f34256bf7a2855c967f65","analyzedAt":"2026-08-30T21:32:31.579Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}