gocolly/colly · warning

Not following redirect to %q: %w

Error message

Not following redirect to %q: %w

What it means

colly wraps any error returned by its redirect-check filter chain with "Not following redirect to %q: %w" inside the Collector's http.Client CheckRedirect func. It means colly deliberately refuses to follow an HTTP redirect because the redirect target (URL or host) fails URLFilters/DomainFilters or other allowed-domain checks. The wrapped inner error carries the actual reason (e.g. 'Forbidden domain').

Source

Thrown at colly.go:1464

		debugger:               c.debugger,
		Async:                  c.Async,
		redirectHandler:        c.redirectHandler,
		errorCallbacks:         make([]ErrorCallback, 0, 8),
		htmlCallbacks:          make([]*htmlCallbackContainer, 0, 8),
		xmlCallbacks:           make([]*xmlCallbackContainer, 0, 8),
		scrapedCallbacks:       make([]ScrapedCallback, 0, 8),
		lock:                   c.lock,
		requestCallbacks:       make([]RequestCallback, 0, 8),
		responseCallbacks:      make([]ResponseCallback, 0, 8),
		robotsMap:              c.robotsMap,
		wg:                     &sync.WaitGroup{},
	}
}

func (c *Collector) checkRedirectFunc() func(req *http.Request, via []*http.Request) error {
	return func(req *http.Request, via []*http.Request) error {
		if err := c.checkFilters(req.URL.String(), req.URL.Hostname()); err != nil {
			return fmt.Errorf("Not following redirect to %q: %w", req.URL, err)
		}

		// Page may set cookies and respond with a redirect to itself.
		// Some example of such redirect "cycles":
		//
		// example.com -(set cookie)-> example.com
		// example.com -> auth.example.com -(set cookie)-> example.com
		// www.example.com -> example.com -(set cookie)-> example.com
		//
		// We must not return "already visited" error in such cases.
		// So ignore redirect cycles when checking for URL revisit.
		redirectCycle := false
		normalizedURL := normalizeURL(req.URL.String())
		for _, viaReq := range via {
			viaURL := normalizeURL(viaReq.URL.String())
			if viaURL == normalizedURL {
				redirectCycle = true
				break

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Add the redirect target domain (and variants like www./CDN hosts) to Collector.AllowedDomains
2. Relax URLFilters that exclude the redirect target URL
3. Inspect the wrapped inner error (%w) to confirm which filter rejected the URL
4. If you intend to follow all redirects, clear over-restrictive domain filters

Example fix

// before
c := colly.NewCollector(colly.AllowedDomains("example.com"))
// after (site redirects via www. and its CDN)
c := colly.NewCollector(colly.AllowedDomains("example.com", "www.example.com", "cdn.example.com"))

// before visiting, check the URL is allowed
c := colly.NewCollector(colly.AllowedDomains("example.com", "www.example.com"))
if len(c.AllowedDomains) > 0 && !containsDomain(c.AllowedDomains, targetHost()) {
    // target would be rejected on redirect
}

var redirectErr *colly.RedirectedError // if surfaced via c.OnError
func isRedirectFilterErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "Not following redirect to")
}

c.OnError(func(r *colly.Response, err error) {
    if strings.Contains(err.Error(), "Not following redirect to") {
        log.Printf("redirect blocked to %s: %v", r.Request.URL, err) // inspect wrapped cause
        return
    }
    panic(err)
})

When it happens

Trigger: A request receives a 3xx redirect and the redirect target URL/host fails c.checkFilters: the target domain is not in AllowedDomains / not allowed by URLFilters, or the redirect chain exceeds depth limits. Happens with c.Visit, forms, or any request when the server redirects off-domain.

Common situations: Scraping sites that redirect to a login/SSO or CDN host not listed in AllowedDomains; http->https or www/non-www redirects to an unallowed variant; sites redirecting to an error page on another domain; configuring AllowedDomains too narrowly.

Related errors

• ErrForbiddenDomain
• ErrMissingURL
• ErrMaxDepth
• ErrForbiddenURL
• ErrNoURLFiltersMatch

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/c5d021e473da0940. Report an issue: GitHub.