What causes "failed to read response body" in Netflix/chaosmonkey?

Check the log line "Unexpected response: %d" printed just before this error for the actual status code.

failed to read response body

Error message

failed to read response body

What it means

When the Spinnaker task POST returns a non-200 status, Execute tries to read the error response body to include in a more detailed message. This error is thrown if that ioutil.ReadAll fails, meaning chaosmonkey saw a failed HTTP status but could not even read the error body. The status code was already logged via log.Printf, but the detailed body-based error is lost.

Source

Thrown at spinnaker/terminator.go:97

	}

	payload := killJSONPayload(ins, otherID, s.user)
	resp, err := s.client.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		return errors.Wrap(err, fmt.Sprintf("POST to %s failed, (body '%s')", url, string(payload)))
	}

	defer func() {
		if cerr := resp.Body.Close(); cerr != nil && err == nil {
			err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body of %s", url))
		}
	}()

	if resp.StatusCode != http.StatusOK {
		log.Printf("Unexpected response: %d", resp.StatusCode)
		contents, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			return errors.Wrap(err, "failed to read response body")
		}
		return fmt.Errorf("unexpected response code: %d, body: %s", resp.StatusCode, string(contents))
	}

	return nil
}

// killJsonPayload generates the JSON request body for terminating an instance
// otherID is an optional second instance ID, as some backends may have a second
// identifer.
func killJSONPayload(ins chaosmonkey.Instance, otherID string, spinnakerUser string) []byte {
	var desc string
	if otherID != "" {
		desc = fmt.Sprintf("Chaos Monkey terminate instance: %s %s (%s, %s, %s)", ins.ID(), otherID, ins.AccountName(), ins.RegionName(), ins.ASGName())
	} else {
		desc = fmt.Sprintf("Chaos Monkey terminate instance: %s (%s, %s, %s)", ins.ID(), ins.AccountName(), ins.RegionName(), ins.ASGName())
	}

View on GitHub (pinned to eaa28fb761)

Solutions

Check the log line "Unexpected response: %d" printed just before this error for the actual status code.
Read and drain the body with a bounded reader (io.LimitReader) and tolerate partial reads to keep the status-based error path robust.
Investigate why Spinnaker rejected the task (permissions, invalid payload, Gate errors) using Gate logs.
Retry the termination if the status suggests a transient gateway error (502/503/504).
Confirm chaosmonkey's Spinnaker credentials/roles allow task creation for the target application.

Example fix

// before
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
	return errors.Wrap(err, "failed to read response body")
}
// after
contents, readErr := ioutil.ReadAll(io.LimitReader(resp.Body, 1<<20))
if readErr != nil {
	return fmt.Errorf("unexpected response code: %d, (body read failed: %v)", resp.StatusCode, readErr)
}

Defensive patterns

Strategy: fallback

Validate before calling

if resp.StatusCode != http.StatusOK {
	contents, readErr := ioutil.ReadAll(io.LimitReader(resp.Body, 1<<20))
	if readErr != nil {
		// fall back to the status code alone
		return fmt.Errorf("unexpected response code: %d (body unavailable)", resp.StatusCode)
	}
	return fmt.Errorf("unexpected response code: %d, body: %s", resp.StatusCode, string(contents))
}

Try / catch

if err != nil {
	// fall back to status-code-only error; the code was already logged
	log.Printf("failed to read response body: %v (status was %d)", err, resp.StatusCode)
	return fmt.Errorf("unexpected response code: %d", resp.StatusCode)
}

Prevention

Always include the HTTP status in errors even when the body can't be read
Drain bodies with io.LimitReader to bound memory and tolerate truncation
Watch the "Unexpected response: %d" log for Gate rejections (auth, invalid payload)
Retry on 502/503/504 gateway statuses

When it happens

Trigger: POST <spinnaker-api>/tasks returned a status other than 200, and then reading resp.Body failed with a non-EOF error — connection reset while reading the error response, or a truncated chunked error response.

Common situations: Gate or an LB rejecting the kill request (4xx/5xx) and dropping the connection before the error body completed; auth proxies terminating connections on authorization failures; Gate crash mid-response while rejecting a task.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/f0eb3283cdbdb3ac. Report an issue: GitHub.