projectdiscovery/subfinder · error

error reading response body

Error message

error reading response body

What it means

The Netlas source reads the full HTTP response body with io.ReadAll after fetching the domains count; if that read fails it emits this generic error on the results channel and stops. Note the underlying err is swallowed — the message does not include it, so only the fact that body reading failed is reported. The request succeeded at HTTP level but the body could not be fully transferred.

Solutions

1. Retry the enumeration — this is usually a transient network interruption
2. Wrap the error with %w so the real cause (reset, timeout, EOF) is visible for diagnosis
3. Increase HTTP client timeout if large Netlas responses are being cut off
4. Check network path stability: proxies, VPN, and DNS reliability

Example fix

// before
results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("error reading response body")}
// after
results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("error reading response body: %w", err)}

if _, err := http.Get("https://api.netlas.io"); err != nil {
    return fmt.Errorf("netlas API unreachable, check network/API key before running: %w", err)
}

for res := range src.Fetch(ctx, domain) {
    if res.Type == subscraping.Error {
        if errors.Is(res.Error, io.ErrUnexpectedEOF) || isTransientNetErr(res.Error) {
            retryFetch() // body read interrupted — retry
            continue
        }
        log.Printf("netlas: %v", res.Error)
    }
}

When it happens

Trigger: io.ReadAll(resp1.Body) returns an error: connection reset or timeout while streaming the body, server closing the connection early, or TLS/transport interruption mid-response during the Netlas count query or subsequent fetch.

Common situations: Flaky network or VPN drops mid-response, aggressive timeouts cutting off a large JSON body, proxies/CDNs closing keep-alive connections prematurely.

Related errors

• request failed with status
• unexpected status code
• %s
• request failed with status
• unexpected status code

AI-assisted analysis of projectdiscovery/subfinder@7a0b91f0fa (2026-09-06). Data as JSON: /api/errors/fb04b5d8d5c7fe8a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/subscraping/sources/netlas/netlas.go:92

			"X-API-Key": randomApiKey,
		}, nil, subscraping.BasicAuth{})

		if err != nil {
			results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
			s.errors++
			session.DiscardHTTPResponse(resp1)
			return
		}
		defer func() {
			if err := resp1.Body.Close(); err != nil {
				results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
				s.errors++
			}
		}()

		body, err := io.ReadAll(resp1.Body)
		if err != nil {
			results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf("error reading response body")}
			s.errors++
			return
		}

		// Parse the JSON response
		var domainsCount DomainsCountResponse
		err = json.Unmarshal(body, &domainsCount)
		if err != nil {
			results <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}
			s.errors++
			return
		}

		// Make a single POST request to get all domains via download method

		apiUrl := "https://app.netlas.io/api/domains/download/"
		query := fmt.Sprintf("domain:*.%s AND NOT domain:%s", domain, domain)
		requestBody := map[string]any{

View on GitHub (pinned to 7a0b91f0fa)