What causes "webscan_response_body_process_failed: %w" in shadow1ng/fscan?

Unwrap the error to find the underlying read/decompression failure.

webscan_response_body_process_failed

webscan_response_body_process_failed

Error message

webscan_response_body_process_failed: %w

What it means

ParseResponse reads the HTTP response body via getRespBody while converting an http.Response into the scanner's response struct. If reading the body fails, the error is wrapped as webscan_response_body_process_failed. Called from DoRequest, so any failed POC response body read surfaces through this error.

Source

Thrown at webscan/lib/Eval.go:651

	} else {
		respURL = &UrlType{}
	}
	resp := Response{
		Status:      int32(oResp.StatusCode),
		URL:         respURL,
		Headers:     make(map[string]string),
		ContentType: oResp.Header.Get("Content-Type"),
	}

	// 复制响应头，合并多值头部为分号分隔的字符串
	for k := range oResp.Header {
		resp.Headers[k] = strings.Join(oResp.Header.Values(k), ";")
	}

	// 读取并解析响应体
	body, err := getRespBody(oResp)
	if err != nil {
		return nil, fmt.Errorf("%s: %w", i18n.GetText("webscan_response_body_process_failed"), err)
	}
	resp.Body = body

	return &resp, nil
}

// getRespBody 读取 HTTP 响应体并处理可能的 gzip 压缩
func getRespBody(oResp *http.Response) ([]byte, error) {
	// 读取原始响应体
	body, err := io.ReadAll(io.LimitReader(oResp.Body, maxPOCResponseBodyBytes))
	if err != nil && !errors.Is(err, io.EOF) && len(body) == 0 {
		return nil, err
	}

	// 处理 gzip 压缩
	if strings.Contains(oResp.Header.Get("Content-Encoding"), "gzip") {
		reader, err := gzip.NewReader(bytes.NewReader(body))
		if err != nil {

View on GitHub (pinned to 95cc12e753)

Solutions

Unwrap the error to find the underlying read/decompression failure.
Retry the request; transient connection resets during body reads are common and the scan state already counts TCP success/failure.
Increase read timeouts so the body stream is not cut off mid-transfer.
Check whether the response uses an encoding (gzip/brotli) that getRespBody handles.

Example fix

// before
resp, err := DoRequest(req)
if err != nil { return err }

// after
resp, err := DoRequest(req)
if err != nil {
    // includes webscan_response_body_process_failed from body reads
    return fmt.Errorf("poc request failed: %w", err)
}

Defensive patterns

Strategy: try-catch

Validate before calling

if oResp.StatusCode == 0 || oResp.Body == nil {
    return errors.New("response has no readable body")
}

Try / catch

parsed, err := DoRequest(req)
if err != nil {
    if strings.Contains(err.Error(), "webscan_response_body_process_failed") {
        // retry once; body reads often fail transiently
    }
    return err
}

Prevention

Set generous read timeouts so body streams aren't cut mid-transfer.
Close response bodies only after full parsing (DoRequest already defers Body.Close).
Retry on connection-reset style failures.

When it happens

Trigger: getRespBody(oResp) returns an error: response body reader errors mid-read (connection reset while streaming body), truncated/chunked transfer errors, or a body already closed before parsing.

Common situations: Target closes the connection before the full body arrives; gzip-encoded bodies that fail to decompress inside getRespBody; very large responses hitting read limits; flaky mobile/GMTLS connections.

Understand the failure class

Background: "Invalid JSON response" and "Failed to parse response" errors: when an API answers 200 but the body isn't the JSON your library expected — this error's family across 28 libraries.

Related errors

AI-assisted analysis of shadow1ng/fscan@95cc12e753 (2026-09-06). Data as JSON: /api/errors/e3822374688c179f. Report an issue: GitHub.