What causes "body read failed at %s" in Netflix/chaosmonkey?

Retry the Get request — transient body truncation usually succeeds on retry

body read failed at %s

Error message

body read failed at %s

What it means

Once a 200 response is received, Get reads the entire response body with ioutil.ReadAll. A read failure (connection reset mid-body, premature close, chunked encoding error) is wrapped as 'body read failed at <url>'.

Source

Thrown at spinnaker/config.go:48

	resp, err := s.client.Get(url)
	if err != nil {
		return nil, errors.Wrapf(err, "http get failed at %s", url)
	}

	defer func() {
		if cerr := resp.Body.Close(); cerr != nil && err == nil {
			err = errors.Wrapf(err, "body close failed at %s", url)
		}
	}()

	// should return a 200
	if resp.StatusCode != http.StatusOK {
		return nil, errors.Errorf("unexpected response code (%d) from %s", resp.StatusCode, url)
	}

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return nil, errors.Wrapf(err, "body read failed at %s", url)
	}

	return fromJSON(body)
}

View on GitHub (pinned to eaa28fb761)

Solutions

Retry the Get request — transient body truncation usually succeeds on retry
Check proxy/load balancer timeout settings between the monkey and the gate
Inspect the wrapped cause for whether the connection was reset or timed out
Consider a client with sane timeouts and retry policy for the gate client

Example fix

// caller-side retry guard
var cfg *chaosmonkey.AppConfig
var err error
for i := 0; i < 3; i++ {
    cfg, err = getter.Get(app)
    if err == nil || !strings.Contains(fmt.Sprintf("%v", err), "body read failed") {
        break
    }
    time.Sleep(time.Duration(1<<i) * time.Second)
}

Defensive patterns

Strategy: retry

Validate before calling

// check content-length/type hints before reading fully
ct := resp.Header.Get("Content-Type")
if !strings.HasPrefix(ct, "application/json") {
    return fmt.Errorf("expected JSON, got %s", ct)
}

Try / catch

cfg, err := getter.Get(app)
if err != nil && strings.Contains(err.Error(), "body read failed") {
    time.Sleep(2 * time.Second)
    cfg, err = getter.Get(app) // transient truncations usually clear on retry
}

Prevention

Increase proxy/load balancer response timeouts for the gate route
Set reasonable client timeouts so partial reads fail fast and retry cleanly
Monitor for connection resets between the monkey and the gate
Avoid extremely large responses by keeping Spinnaker app configs lean

When it happens

Trigger: ioutil.ReadAll(resp.Body) errors: connection dropped while streaming the body, proxy/gateway terminating the response early, TLS errors mid-transfer, or body already closed by intermediate handling.

Common situations: Large app configs over flaky networks; proxies/load balancers with short idle timeouts cutting the response; Spinnaker gate crash mid-response; aggressive keep-alive timeouts.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/c14df82b406531f8. Report an issue: GitHub.