{"record":{"id":"a4b2b7cbd39c6278","repo":"projectdiscovery/subfinder","slug":"request-failed-with-status-d-s","errorCode":null,"errorMessage":"request failed with status %d: %s","messagePattern":"request failed with status (.+?): (.+?)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/subscraping/sources/merklemap/merklemap.go","lineNumber":119,"sourceCode":"\n\t}\n}\n\n// fetchPage fetches a single page of results\nfunc (s *Source) fetchPage(ctx context.Context, baseURL string, page int, headers map[string]string, session *subscraping.Session) (*response, error) {\n\turl := baseURL + \"&page=\" + strconv.Itoa(page)\n\n\ts.requests++\n\tresp, err := session.Get(ctx, url, \"\", headers)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer session.DiscardHTTPResponse(resp)\n\n\tif resp.StatusCode != 200 {\n\t\trespBody, err := io.ReadAll(resp.Body)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"request failed with status %d: %s\", resp.StatusCode, err)\n\t\t}\n\t\treturn nil, fmt.Errorf(\"request failed with status %d: %s\", resp.StatusCode, string(respBody))\n\t}\n\n\tvar pageResponse response\n\trespBody, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tdecoder := json.NewDecoder(bytes.NewReader(respBody))\n\tif err := decoder.Decode(&pageResponse); err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &pageResponse, nil\n}\n","sourceCodeStart":101,"sourceCodeEnd":137,"githubUrl":"https://github.com/projectdiscovery/subfinder/blob/7a0b91f0fac01b62c65328bd771a3560ae611d6a/pkg/subscraping/sources/merklemap/merklemap.go#L101-L137","documentation":"In MerkleMap's fetchPage, when the HTTP status is not 200 the code tries to read the response body to include it in the error; if io.ReadAll itself fails, the status code and the read error are wrapped into this message and returned up to fetchAllPages. This is the rare I/O-failure branch of the non-200 handling — the body could not be read (e.g. connection reset mid-read, body already closed).","triggerScenarios":"MerkleMap API returns a non-200 status AND reading the error body fails: connection dropped while reading resp.Body, body already consumed/closed, or a transport error during ReadAll (err != nil branch).","commonSituations":"Unstable network causing truncated responses, an intermediary/proxy closing the connection on an error response, or timeouts killing the connection before the body could be drained.","solutions":["Retry the request — this branch reflects a transient network/IO problem, not an API answer","Check network stability, proxies, and timeouts configured on the HTTP client","If it persists, inspect whether the MerkleMap endpoint URL is reachable at all (curl -v)","Verify the go-ipfs/certificate or proxy stack is not closing connections prematurely"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"resp, err := http.Head(\"https://merklemap.com\")\nif err != nil {\n    return fmt.Errorf(\"merklemap unreachable, check network before running: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"for res := range src.Fetch(ctx, domain) {\n    if res.Type == subscraping.Error {\n        if errors.Is(res.Error, context.DeadlineExceeded) || isTransientNetErr(res.Error) {\n            // retry the page fetch with backoff\n            continue\n        }\n        log.Printf(\"merklemap: %v\", res.Error)\n    }\n}","preventionTips":["Run over a stable network; avoid flaky VPNs/proxies for long enumeration runs","Set generous HTTP timeouts so large bodies are not truncated mid-read","Treat IO-failure errors as transient and implement bounded retries","Verify endpoint reachability with curl -v when errors persist"],"tags":["network","http","io","status-code"],"backgroundTag":"http-non-200-response","analyzedSha":"7a0b91f0fac01b62c65328bd771a3560ae611d6a","analyzedAt":"2026-09-06T23:52:02.109Z","contentChangedAt":"2026-09-06T23:52:02.109Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}