What causes "http.StatusText(response.StatusCode)" in gocolly/colly?

Check response.StatusCode inside OnError to distinguish status classes

http.StatusText(response.StatusCode)

Error message

http.StatusText(response.StatusCode)

What it means

In Collector.handleOnError, when the error is nil but the response status code is >= 300 and ParseHTTPErrorResponse is false, colly synthesizes err = errors.New(http.StatusText(response.StatusCode)) and invokes OnError. The message is just the standard status text (e.g. "Not Found", "Internal Server Error"). This is colly's default way of treating HTTP error statuses as errors.

Solutions

Check response.StatusCode inside OnError to distinguish status classes
Set c.ParseHTTPErrorResponse = true if you want error-status responses delivered to OnResponse instead
Inspect the target URL for the cause of the 4xx/5xx (auth, robots, bot detection)
Use errors.Is/inspection of the message against http.StatusText codes to branch your handling

Example fix

// before: default collector, error pages go to OnError
c := colly.NewCollector()
// after: receive error responses in OnResponse
c := colly.NewCollector()
c.ParseHTTPErrorResponse = true

Defensive patterns

Strategy: try-catch

Validate before calling

// preflight: decide policy before crawling
c := colly.NewCollector()
c.ParseHTTPErrorResponse = true // receive >=300 responses in OnResponse instead

Type guard

func isHTTPStatusError(err error) bool {
    if err == nil { return false }
    return http.StatusText(http.StatusTextLength(len(err.Error()))) != "" &&
        slices.ContainsFunc(statusTexts(), func(s string) bool { return err.Error() == s })
}

Try / catch

c.OnError(func(r *colly.Response, err error) {
    if r.StatusCode >= 400 {
        log.Printf("HTTP %d (%s): %s", r.StatusCode, err.Error(), r.Request.URL)
        return
    }
    log.Printf("transport error: %v", err)
})

Prevention

Always implement OnError to inspect r.StatusCode explicitly
Set ParseHTTPErrorResponse deliberately per crawler use case
Log the URL and status together for triage
Don't string-match the error text; branch on StatusCode

When it happens

Trigger: Any c.Visit/request receiving a response with StatusCode >= 300 while ParseHTTPErrorResponse is false (the default), causing OnError with a status-text error.

Common situations: Hitting 404/403/500 pages during crawls; sites returning 3xx redirects the backend did not follow; developers confused why OnError fires with a plain-English message instead of a network error.

Related errors

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/db899d283dda4350. Report an issue: GitHub.

Appendix: source

Thrown at colly.go:1324

				if c.debugger != nil {
					c.debugger.Event(createEvent("xml", resp.Request.ID, c.ID, map[string]string{
						"selector": cc.Query,
						"url":      resp.Request.URL.String(),
					}))
				}
				cc.Function(e)
			})
		}
	}
	return nil
}

func (c *Collector) handleOnError(response *Response, err error, request *Request, ctx *Context) error {
	if err == nil && (c.ParseHTTPErrorResponse || response.StatusCode < 300) {
		return nil
	}
	if err == nil && response.StatusCode >= 300 {
		err = errors.New(http.StatusText(response.StatusCode))
	}
	if response == nil {
		response = &Response{
			Request: request,
			Ctx:     ctx,
		}
	}
	if c.debugger != nil {
		c.debugger.Event(createEvent("error", request.ID, c.ID, map[string]string{
			"url":    request.URL.String(),
			"status": http.StatusText(response.StatusCode),
		}))
	}
	if response.Request == nil {
		response.Request = request
	}
	if response.Ctx == nil {
		response.Ctx = request.Ctx

View on GitHub (pinned to 17d1d6ca92)