gocolly/colly · error

ErrNoURLFiltersMatch

ErrNoURLFiltersMatch

Error message

No URLFilters match

What it means

ErrNoURLFiltersMatch is returned when a URL to visit matches none of the collector's URLFilters rules. When URLFilters are configured, they act as a whitelist: a URL that satisfies no rule is rejected. This differs from ErrForbiddenURL in that no rule explicitly disallowed the URL — it simply failed to match any allow rule.

Source

Thrown at colly.go:234

// The prefix for environment variables of Colly settings
const envVariablePrefix = "COLLY_"

var (
	// ErrForbiddenDomain is the error thrown if visiting
	// a domain which is not allowed in AllowedDomains
	ErrForbiddenDomain = errors.New("Forbidden domain")
	// ErrMissingURL is the error type for missing URL errors
	ErrMissingURL = errors.New("Missing URL")
	// ErrMaxDepth is the error type for exceeding max depth
	ErrMaxDepth = errors.New("Max depth limit reached")
	// ErrForbiddenURL is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrForbiddenURL = errors.New("ForbiddenURL")

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list
	ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
	// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
	// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedBeforeRequest = errors.New("Aborted before Do Request")
	// ErrQueueFull is the error returned when the queue is full
	ErrQueueFull = errors.New("Queue MaxSize reached")
	// ErrMaxRequests is the error returned when exceeding max requests
	ErrMaxRequests = errors.New("Max Requests limit reached")
	// ErrRetryBodyUnseekable is the error when retry with not seekable body
	ErrRetryBodyUnseekable = errors.New("Retry Body Unseekable")

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Add a rule to c.URLFilters that matches the URL you want to visit
2. Test the failing URL against each filter rule/regex to find the non-match
3. Relax or remove URLFilters if URL-level whitelisting is not needed
4. Normalize the URL (scheme, query, trailing slash) before Visit so it matches the rules

Example fix

// before
c := colly.NewCollector()
c.URLFilters = []*regexp.Regexp{regexp.MustCompile(`^https://example\.com/blog/\d+$`)}
c.Visit("https://example.com/blog/post/abc") // no rule matches
// after
c := colly.NewCollector()
c.URLFilters = []*regexp.Regexp{regexp.MustCompile(`^https://example\.com/blog/[\w-]+$`)}
c.Visit("https://example.com/blog/post/abc")

for _, r := range filters {
    if r.MatchString(target) {
        break
    }
    // if loop completes, no rule matches -> would trigger ErrNoURLFiltersMatch
}

if err := c.Visit(target); err != nil && errors.Is(err, colly.ErrNoURLFiltersMatch) {
    log.Printf("no URLFilters rule matches %s", target)
    return nil
}

When it happens

Trigger: Calling c.Visit() on a URL when c.URLFilters is populated but no compiled rule matches the URL; mixing URLFilter (single func) and URLFilters (rule list) where the rule list is the active check; rules written for a different path/scheme than the URLs actually encountered.

Common situations: Setting c.URLFilters with pattern rules for one site layout and crawling a differently-shaped URL; regex anchors (^/$) that fail on query strings or trailing slashes; copying filter config between projects whose URL spaces differ.

Related errors

• ErrForbiddenURL
• ErrForbiddenDomain
• ErrMissingURL
• ErrMaxDepth
• ErrRobotsTxtBlocked

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/c53cc6944c528f08. Report an issue: GitHub.