gocolly/colly · warning

ErrAbortedBeforeRequest

ErrAbortedBeforeRequest

Error message

Aborted before Do Request

What it means

ErrAbortedBeforeRequest is returned by the HTTP backend's Do method (http_backend.go) when a pre-flight request-header check returns false — i.e. a checkRequestHeaders callback (installed by the collector) aborted the transfer before the request was sent. It signals that the request never left the client; the abort happened in user code via a header-check hook. The error variable is declared in colly.go:246 alongside the other sentinel errors.

Source

Thrown at colly.go:246

	// ErrForbiddenURL is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrForbiddenURL = errors.New("ForbiddenURL")

	// ErrNoURLFiltersMatch is the error thrown if visiting
	// a URL which is not allowed by URLFilters
	ErrNoURLFiltersMatch = errors.New("No URLFilters match")
	// ErrRobotsTxtBlocked is the error type for robots.txt errors
	ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
	// ErrNoCookieJar is the error type for missing cookie jar
	ErrNoCookieJar = errors.New("Cookie jar is not available")
	// ErrNoPattern is the error type for LimitRules without patterns
	ErrNoPattern = errors.New("No pattern defined in LimitRule")
	// ErrEmptyProxyURL is the error type for empty Proxy URL list
	ErrEmptyProxyURL = errors.New("Proxy URL list is empty")
	// ErrAbortedAfterHeaders is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedAfterHeaders = errors.New("Aborted after receiving response headers")
	// ErrAbortedBeforeRequest is the error returned when OnResponseHeaders aborts the transfer.
	ErrAbortedBeforeRequest = errors.New("Aborted before Do Request")
	// ErrQueueFull is the error returned when the queue is full
	ErrQueueFull = errors.New("Queue MaxSize reached")
	// ErrMaxRequests is the error returned when exceeding max requests
	ErrMaxRequests = errors.New("Max Requests limit reached")
	// ErrRetryBodyUnseekable is the error when retry with not seekable body
	ErrRetryBodyUnseekable = errors.New("Retry Body Unseekable")
)

var envMap = map[string]func(*Collector, string){
	"ALLOWED_DOMAINS": func(c *Collector, val string) {
		c.AllowedDomains = strings.Split(val, ",")
	},
	"CACHE_DIR": func(c *Collector, val string) {
		c.CacheDir = val
	},
	"DETECT_CHARSET": func(c *Collector, val string) {
		c.DetectCharset = isYesString(val)
	},

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Inspect your request-header check callback and make sure returning false is intentional for the request in question
2. Log the request URL/headers inside the callback to see which request is being aborted
3. If the request should proceed, fix the callback condition so it returns true
4. Compare the error with errors.Is(err, colly.ErrAbortedBeforeRequest) to handle aborts distinctly from network failures

Example fix

// before: callback aborts everything
func(r *http.Request) bool { return strings.HasPrefix(r.Header.Get("Authorization"), "Bearer") }
// after: only abort when auth is required
func(r *http.Request) bool { return !strings.Contains(r.URL.Host, "public.example.com") }

// ensure the header-check callback admits this request before sending
if !headerCheckAllows(req) {
    // skip or log instead of letting the backend abort
}

func isAbortedBeforeRequest(err error) bool { return errors.Is(err, colly.ErrAbortedBeforeRequest) }

if err := c.Visit(url); err != nil {
    if errors.Is(err, colly.ErrAbortedBeforeRequest) {
        log.Printf("request aborted by header check: %s", url)
        return nil // intentional skip
    }
    return err
}

When it happens

Trigger: A checkRequestHeaders callback (set through the collector's request-header check hook) returns false, causing http_backend.Do to return ErrAbortedBeforeRequest instead of performing the HTTP request.

Common situations: Developers install a header-check callback to filter out requests (e.g. dropping requests missing an auth header or to certain hosts) and forget that returning false surfaces as this error; it is also hit when callback logic aborts unintentionally on edge cases like empty URLs or stripped headers.

Related errors

• ErrForbiddenDomain
• ErrMissingURL
• ErrMaxDepth
• ErrForbiddenURL
• ErrNoURLFiltersMatch

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/f547206ba0ec7b33. Report an issue: GitHub.