{"record":{"id":"3c6a5627206c2a0e","repo":"projectdiscovery/subfinder","slug":"unexpected-status-code-d-received-from-s","errorCode":null,"errorMessage":"unexpected status code %d received from %s","messagePattern":"unexpected status code (.+?) received from (.+?)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/subscraping/agent.go","lineNumber":177,"sourceCode":"func httpRequestWrapper(client *http.Client, request *http.Request, maxResponseBodySize int64) (*http.Response, error) {\n\tresponse, err := client.Do(request)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Optionally cap the untrusted response body centrally so every source\n\t// (and the debug-logging path below) inherits the same bound when enabled.\n\tLimitResponseBody(response, maxResponseBodySize)\n\n\tif response.StatusCode != http.StatusOK {\n\t\trequestURL, _ := url.QueryUnescape(request.URL.String())\n\n\t\tgologger.Debug().MsgFunc(func() string {\n\t\t\tbuffer := new(bytes.Buffer)\n\t\t\t_, _ = buffer.ReadFrom(response.Body)\n\t\t\treturn fmt.Sprintf(\"Response for failed request against %s:\\n%s\", requestURL, buffer.String())\n\t\t})\n\t\treturn response, fmt.Errorf(\"unexpected status code %d received from %s\", response.StatusCode, requestURL)\n\t}\n\treturn response, nil\n}\n","sourceCodeStart":159,"sourceCodeEnd":181,"githubUrl":"https://github.com/projectdiscovery/subfinder/blob/7a0b91f0fac01b62c65328bd771a3560ae611d6a/pkg/subscraping/agent.go#L159-L181","documentation":"httpRequestWrapper treats HTTP responses with unexpected (non-success) status codes as failures and returns this error along with the response. The wrapper logs the failed response body at debug level, so the error signals a bad upstream status during source scraping.","triggerScenarios":"Any source HTTP request (via HTTPRequest or TestHTTPRequestWrapper) that returns a status code outside the accepted range — e.g. 403 for missing/invalid API keys, 429 rate limiting, 5xx server errors.","commonSituations":"Expired or missing API keys for passive sources; exceeding a provider's rate limits; provider temporarily down; IP blocked by the data source.","solutions":["Check the status code in the error: 401/403 → fix API keys, 429 → slow down or add rate limiting, 5xx → retry later","Configure valid API keys for the failing source in the provider config","Apply rate limiting (-rls) for the affected source and retry","Temporarily disable the failing source if its data is not critical"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"resp, err := http.Head(sourceURL)\nif err == nil && resp.StatusCode >= 200 && resp.StatusCode < 300 {\n    // safe to query\n}","typeGuard":null,"tryCatchPattern":"resp, err := agent.HTTPRequest(ctx, req)\nif err != nil {\n    var status int\n    if _, serr := fmt.Sscanf(err.Error(), \"unexpected status code %d\", &status); serr == nil {\n        switch {\n        case status == 429:\n            // backoff and retry\n        case status == 401 || status == 403:\n            // fix API key\n        default:\n            // retry later or skip source\n        }\n    }\n}","preventionTips":["Configure valid API keys for all sources","Apply per-source rate limits","Monitor for 429/5xx and back off automatically"],"tags":["http","network","scraping"],"backgroundTag":"http-error-response","analyzedSha":"7a0b91f0fac01b62c65328bd771a3560ae611d6a","analyzedAt":"2026-09-06T23:52:02.109Z","contentChangedAt":"2026-09-06T23:52:02.109Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}