What causes "request failed with status %d: %s" in projectdiscovery/subfinder?

Retry the request — this branch reflects a transient network/IO problem, not an API answer

request failed with status

Error message

request failed with status %d: %s

What it means

In MerkleMap's fetchPage, when the HTTP status is not 200 the code tries to read the response body to include it in the error; if io.ReadAll itself fails, the status code and the read error are wrapped into this message and returned up to fetchAllPages. This is the rare I/O-failure branch of the non-200 handling — the body could not be read (e.g. connection reset mid-read, body already closed).

Solutions

Retry the request — this branch reflects a transient network/IO problem, not an API answer
Check network stability, proxies, and timeouts configured on the HTTP client
If it persists, inspect whether the MerkleMap endpoint URL is reachable at all (curl -v)
Verify the go-ipfs/certificate or proxy stack is not closing connections prematurely

Defensive patterns

Strategy: retry

Validate before calling

resp, err := http.Head("https://merklemap.com")
if err != nil {
    return fmt.Errorf("merklemap unreachable, check network before running: %w", err)
}

Try / catch

for res := range src.Fetch(ctx, domain) {
    if res.Type == subscraping.Error {
        if errors.Is(res.Error, context.DeadlineExceeded) || isTransientNetErr(res.Error) {
            // retry the page fetch with backoff
            continue
        }
        log.Printf("merklemap: %v", res.Error)
    }
}

Prevention

Run over a stable network; avoid flaky VPNs/proxies for long enumeration runs
Set generous HTTP timeouts so large bodies are not truncated mid-read
Treat IO-failure errors as transient and implement bounded retries
Verify endpoint reachability with curl -v when errors persist

When it happens

Trigger: MerkleMap API returns a non-200 status AND reading the error body fails: connection dropped while reading resp.Body, body already consumed/closed, or a transport error during ReadAll (err != nil branch).

Common situations: Unstable network causing truncated responses, an intermediary/proxy closing the connection on an error response, or timeouts killing the connection before the body could be drained.

Related errors

AI-assisted analysis of projectdiscovery/subfinder@7a0b91f0fa (2026-09-06). Data as JSON: /api/errors/a4b2b7cbd39c6278. Report an issue: GitHub.

Appendix: source

Thrown at pkg/subscraping/sources/merklemap/merklemap.go:119

	}
}

// fetchPage fetches a single page of results
func (s *Source) fetchPage(ctx context.Context, baseURL string, page int, headers map[string]string, session *subscraping.Session) (*response, error) {
	url := baseURL + "&page=" + strconv.Itoa(page)

	s.requests++
	resp, err := session.Get(ctx, url, "", headers)
	if err != nil {
		return nil, err
	}
	defer session.DiscardHTTPResponse(resp)

	if resp.StatusCode != 200 {
		respBody, err := io.ReadAll(resp.Body)
		if err != nil {
			return nil, fmt.Errorf("request failed with status %d: %s", resp.StatusCode, err)
		}
		return nil, fmt.Errorf("request failed with status %d: %s", resp.StatusCode, string(respBody))
	}

	var pageResponse response
	respBody, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil, err
	}

	decoder := json.NewDecoder(bytes.NewReader(respBody))
	if err := decoder.Decode(&pageResponse); err != nil {
		return nil, err
	}

	return &pageResponse, nil
}

View on GitHub (pinned to 7a0b91f0fa)