What causes "body read failed at %s" in Netflix/chaosmonkey?

Retry the GET; transient read errors usually clear on a second attempt.

body read failed at %s

Error message

body read failed at %s

What it means

OtherID throws this when ioutil.ReadAll fails while reading the body of the Spinnaker instance endpoint response. The HTTP response arrived (status checked later), but the body stream could not be fully consumed — the connection broke mid-read or a read error occurred. The instance URL is included to identify the failing request; it aborts the lookup of the instance's alternate (Titus) ID.

Source

Thrown at spinnaker/terminator.go:158

// OtherID returns the alternate instance id of an instance, if it exists
// If there is no alternate instance id, it returns an empty string
// This is used by Titus, where we also report the uuid
func (s Spinnaker) OtherID(ins chaosmonkey.Instance) (otherID string, err error) {
	url := s.instanceURL(ins.AccountName(), ins.RegionName(), ins.ID())
	resp, err := s.client.Get(url)
	if err != nil {
		return "", errors.Wrap(err, fmt.Sprintf("get failed on %s", url))
	}

	defer func() {
		if cerr := resp.Body.Close(); cerr != nil && err == nil {
			err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body from %s", url))
		}
	}()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return "", errors.Wrap(err, fmt.Sprintf("body read failed at %s", url))
	}

	// Example of response body:
	/*
		{
			...
			"health": [
				{
					"type": "Titus",
					"healthClass": "platform",
					"state": "Up"
				},
				{
					"instanceId": "55fe33ab-5b66-450a-85f7-f3129806b87f",
					"titusTaskId": "Titus-123456-worker-0-0",
					...
				}
			],

View on GitHub (pinned to eaa28fb761)

Solutions

Retry the GET; transient read errors usually clear on a second attempt.
Check errors.Cause to distinguish unexpected EOF, connection reset, or timeout, and address the matching layer.
Increase client read timeouts and check LB/proxy response timeouts.
Verify Gate health and scale/latency at the time of failure.
Bound the read with io.LimitReader and consider partial-tolerant parsing if bodies are large.

Defensive patterns

Strategy: retry

Validate before calling

// retry the whole GET if the body read fails; the request is idempotent
var body []byte
for i := 0; i < 3; i++ {
	resp, err := client.Get(url)
	if err != nil {
		continue
	}
	body, err = ioutil.ReadAll(io.LimitReader(resp.Body, 1<<20))
	resp.Body.Close()
	if err == nil {
		break
	}
}
if body == nil {
	return fmt.Errorf("body read failed at %s after retries", url)
}

Try / catch

body, err := ioutil.ReadAll(resp.Body)
if err != nil {
	if ne, ok := errors.Cause(err).(net.Error); ok && ne.Timeout() {
		// retry with longer read timeout
	}
	return "", errors.Wrap(err, fmt.Sprintf("body read failed at %s", url))
}

Prevention

Use io.LimitReader to cap reads and reduce exposure to stalled bodies
Increase client read timeouts if bodies are large or Gate is slow
Check LB/proxy response timeouts vs Gate latency under load
Retry idempotent GETs on transient read errors

When it happens

Trigger: GET <spinnaker-api>/<account>/<region>/<instanceId> returned headers, but reading resp.Body hit a non-EOF error: mid-transfer connection reset, truncated chunked encoding, or a read timeout on a stalled body.

Common situations: Gate under heavy load producing stalled/truncated responses; LB idle timeouts killing slow responses; flaky network between chaosmonkey and Spinnaker; Titus instance lookups of large task metadata being interrupted.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/1e44fef5fcc10d52. Report an issue: GitHub.