{"record":{"id":"db899d283dda4350","repo":"gocolly/colly","slug":"http-statustext-response-statuscode","errorCode":null,"errorMessage":"http.StatusText(response.StatusCode)","messagePattern":"http\\.StatusText\\(response\\.StatusCode\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"colly.go","lineNumber":1324,"sourceCode":"\t\t\t\tif c.debugger != nil {\n\t\t\t\t\tc.debugger.Event(createEvent(\"xml\", resp.Request.ID, c.ID, map[string]string{\n\t\t\t\t\t\t\"selector\": cc.Query,\n\t\t\t\t\t\t\"url\":      resp.Request.URL.String(),\n\t\t\t\t\t}))\n\t\t\t\t}\n\t\t\t\tcc.Function(e)\n\t\t\t})\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (c *Collector) handleOnError(response *Response, err error, request *Request, ctx *Context) error {\n\tif err == nil && (c.ParseHTTPErrorResponse || response.StatusCode < 300) {\n\t\treturn nil\n\t}\n\tif err == nil && response.StatusCode >= 300 {\n\t\terr = errors.New(http.StatusText(response.StatusCode))\n\t}\n\tif response == nil {\n\t\tresponse = &Response{\n\t\t\tRequest: request,\n\t\t\tCtx:     ctx,\n\t\t}\n\t}\n\tif c.debugger != nil {\n\t\tc.debugger.Event(createEvent(\"error\", request.ID, c.ID, map[string]string{\n\t\t\t\"url\":    request.URL.String(),\n\t\t\t\"status\": http.StatusText(response.StatusCode),\n\t\t}))\n\t}\n\tif response.Request == nil {\n\t\tresponse.Request = request\n\t}\n\tif response.Ctx == nil {\n\t\tresponse.Ctx = request.Ctx","sourceCodeStart":1306,"sourceCodeEnd":1342,"githubUrl":"https://github.com/gocolly/colly/blob/17d1d6ca92bd32a5651f34256bf7a2855c967f65/colly.go#L1306-L1342","documentation":"In Collector.handleOnError, when the error is nil but the response status code is >= 300 and ParseHTTPErrorResponse is false, colly synthesizes err = errors.New(http.StatusText(response.StatusCode)) and invokes OnError. The message is just the standard status text (e.g. \"Not Found\", \"Internal Server Error\"). This is colly's default way of treating HTTP error statuses as errors.","triggerScenarios":"Any c.Visit/request receiving a response with StatusCode >= 300 while ParseHTTPErrorResponse is false (the default), causing OnError with a status-text error.","commonSituations":"Hitting 404/403/500 pages during crawls; sites returning 3xx redirects the backend did not follow; developers confused why OnError fires with a plain-English message instead of a network error.","solutions":["Check response.StatusCode inside OnError to distinguish status classes","Set c.ParseHTTPErrorResponse = true if you want error-status responses delivered to OnResponse instead","Inspect the target URL for the cause of the 4xx/5xx (auth, robots, bot detection)","Use errors.Is/inspection of the message against http.StatusText codes to branch your handling"],"exampleFix":"// before: default collector, error pages go to OnError\nc := colly.NewCollector()\n// after: receive error responses in OnResponse\nc := colly.NewCollector()\nc.ParseHTTPErrorResponse = true","handlingStrategy":"try-catch","validationCode":"// preflight: decide policy before crawling\nc := colly.NewCollector()\nc.ParseHTTPErrorResponse = true // receive >=300 responses in OnResponse instead","typeGuard":"func isHTTPStatusError(err error) bool {\n    if err == nil { return false }\n    return http.StatusText(http.StatusTextLength(len(err.Error()))) != \"\" &&\n        slices.ContainsFunc(statusTexts(), func(s string) bool { return err.Error() == s })\n}","tryCatchPattern":"c.OnError(func(r *colly.Response, err error) {\n    if r.StatusCode >= 400 {\n        log.Printf(\"HTTP %d (%s): %s\", r.StatusCode, err.Error(), r.Request.URL)\n        return\n    }\n    log.Printf(\"transport error: %v\", err)\n})","preventionTips":["Always implement OnError to inspect r.StatusCode explicitly","Set ParseHTTPErrorResponse deliberately per crawler use case","Log the URL and status together for triage","Don't string-match the error text; branch on StatusCode"],"tags":["http-status","error-handling","colly"],"backgroundTag":"http-error-status","analyzedSha":"17d1d6ca92bd32a5651f34256bf7a2855c967f65","analyzedAt":"2026-08-30T21:32:31.579Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}