{"record":{"id":"fb04b5d8d5c7fe8a","repo":"projectdiscovery/subfinder","slug":"error-reading-response-body","errorCode":null,"errorMessage":"error reading response body","messagePattern":"error reading response body","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/subscraping/sources/netlas/netlas.go","lineNumber":92,"sourceCode":"\t\t\t\"X-API-Key\": randomApiKey,\n\t\t}, nil, subscraping.BasicAuth{})\n\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp1)\n\t\t\treturn\n\t\t}\n\t\tdefer func() {\n\t\t\tif err := resp1.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tbody, err := io.ReadAll(resp1.Body)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"error reading response body\")}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\t// Parse the JSON response\n\t\tvar domainsCount DomainsCountResponse\n\t\terr = json.Unmarshal(body, &domainsCount)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\t// Make a single POST request to get all domains via download method\n\n\t\tapiUrl := \"https://app.netlas.io/api/domains/download/\"\n\t\tquery := fmt.Sprintf(\"domain:*.%s AND NOT domain:%s\", domain, domain)\n\t\trequestBody := map[string]any{","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/projectdiscovery/subfinder/blob/7a0b91f0fac01b62c65328bd771a3560ae611d6a/pkg/subscraping/sources/netlas/netlas.go#L74-L110","documentation":"The Netlas source reads the full HTTP response body with io.ReadAll after fetching the domains count; if that read fails it emits this generic error on the results channel and stops. Note the underlying err is swallowed — the message does not include it, so only the fact that body reading failed is reported. The request succeeded at HTTP level but the body could not be fully transferred.","triggerScenarios":"io.ReadAll(resp1.Body) returns an error: connection reset or timeout while streaming the body, server closing the connection early, or TLS/transport interruption mid-response during the Netlas count query or subsequent fetch.","commonSituations":"Flaky network or VPN drops mid-response, aggressive timeouts cutting off a large JSON body, proxies/CDNs closing keep-alive connections prematurely.","solutions":["Retry the enumeration — this is usually a transient network interruption","Wrap the error with %w so the real cause (reset, timeout, EOF) is visible for diagnosis","Increase HTTP client timeout if large Netlas responses are being cut off","Check network path stability: proxies, VPN, and DNS reliability"],"exampleFix":"// before\nresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"error reading response body\")}\n// after\nresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"error reading response body: %w\", err)}","handlingStrategy":"retry","validationCode":"if _, err := http.Get(\"https://api.netlas.io\"); err != nil {\n    return fmt.Errorf(\"netlas API unreachable, check network/API key before running: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"for res := range src.Fetch(ctx, domain) {\n    if res.Type == subscraping.Error {\n        if errors.Is(res.Error, io.ErrUnexpectedEOF) || isTransientNetErr(res.Error) {\n            retryFetch() // body read interrupted — retry\n            continue\n        }\n        log.Printf(\"netlas: %v\", res.Error)\n    }\n}","preventionTips":["Retry on body-read failures — they are almost always transient network drops","Increase HTTP client read timeouts so large Netlas JSON bodies complete","Ensure a stable connection: avoid mid-run VPN/proxy drops","Configure a valid NETLAS_API_KEY so responses are not cut off by auth redirects"],"tags":["network","io","http","response-body"],"backgroundTag":"network-request-failed","analyzedSha":"7a0b91f0fac01b62c65328bd771a3560ae611d6a","analyzedAt":"2026-09-06T23:52:02.109Z","contentChangedAt":"2026-09-06T23:52:02.109Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}