What causes "failed to close response body of %s" in Netflix/chaosmonkey?

Treat as mostly benign if the kill succeeded (status 200 was returned); check Gate logs to confirm the task ran.

failed to close response body of %s

Error message

failed to close response body of %s

What it means

This deferred error is raised when resp.Body.Close() returns an error after the POST to the Spinnaker task endpoint — and only when the main flow had no other error (cerr != nil && err == nil). It means the response body could not be closed cleanly, typically indicating an interrupted or abnormal connection state. It replaces the named return value err so the caller still sees a failure even though the termination request itself succeeded.

Source

Thrown at spinnaker/terminator.go:89

// Execute implements term.Terminator.Execute
func (s Spinnaker) Execute(trm chaosmonkey.Termination) (err error) {
	ins := trm.Instance
	url := s.tasksURL(ins.AppName())

	otherID, err := s.OtherID(ins)
	if err != nil {
		return errors.Wrap(err, "retrieve other id failed")
	}

	payload := killJSONPayload(ins, otherID, s.user)
	resp, err := s.client.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		return errors.Wrap(err, fmt.Sprintf("POST to %s failed, (body '%s')", url, string(payload)))
	}

	defer func() {
		if cerr := resp.Body.Close(); cerr != nil && err == nil {
			err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body of %s", url))
		}
	}()

	if resp.StatusCode != http.StatusOK {
		log.Printf("Unexpected response: %d", resp.StatusCode)
		contents, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			return errors.Wrap(err, "failed to read response body")
		}
		return fmt.Errorf("unexpected response code: %d, body: %s", resp.StatusCode, string(contents))
	}

	return nil
}

// killJsonPayload generates the JSON request body for terminating an instance
// otherID is an optional second instance ID, as some backends may have a second
// identifer.

View on GitHub (pinned to eaa28fb761)

Solutions

Treat as mostly benign if the kill succeeded (status 200 was returned); check Gate logs to confirm the task ran.
Read/drain the body fully (ioutil.ReadAll) before Close if the flow doesn't, to make connection reuse clean.
Disable keepalive or close idle connections if resets recur (http.Transport settings).
Inspect errors.Cause for the specific close error and correlate with network/LB logs.
Upgrade Go or adjust http.Transport (IdleConnTimeout, MaxIdleConnsPerHost) to reduce stale keepalive connections.

Example fix

// before
defer func() {
	if cerr := resp.Body.Close(); cerr != nil && err == nil {
		err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body of %s", url))
	}
}()
// after
defer func() {
	io.Copy(io.Discard, resp.Body) // drain before close
	if cerr := resp.Body.Close(); cerr != nil && err == nil {
		err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body of %s", url))
	}
}()

Defensive patterns

Strategy: try-catch

Validate before calling

// drain the body before the deferred close to reduce close errors
resp, err := client.Post(url, "application/json", bytes.NewReader(payload))
if err != nil {
	return err
}
io.Copy(io.Discard, resp.Body)

Try / catch

err := terminateInstance(payload)
if err != nil {
	if strings.Contains(err.Error(), "failed to close response body") && killConfirmed(taskID) {
		log.Printf("ignoring close error; kill task %s confirmed", taskID)
		return nil
	}
	return err
}

Prevention

Always read/drain resp.Body before Close for clean connection reuse
Tune http.Transport keepalive settings (IdleConnTimeout) to avoid stale connections
Confirm task success via Spinnaker task status before trusting close-error failures
Log close errors at warning level; they rarely invalidate the termination

When it happens

Trigger: The POST to <spinnaker-api>/tasks received a response, but calling Close() on resp.Body returned a non-nil error — usually because the underlying connection was already broken/reset while the body was being torn down.

Common situations: Keepalive connection reset by Gate or an intermediate LB at body-close time; client timeouts racing with response draining; Go http.Client reusing a connection that the server closed; reading very large error-response bodies earlier in the flow interacting badly with connection state.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/b8fb4e14521e1ccd. Report an issue: GitHub.