What causes "failed to close response body from %s" in Netflix/chaosmonkey?

Check whether the underlying data was still correct; if so, this is a teardown artifact — drain the body before closing to reduce it.

Netflix/chaosmonkey · warning

failed to close response body from %s

Error message

failed to close response body from %s

What it means

This deferred error fires when resp.Body.Close() fails inside OtherID, but only when the rest of OtherID completed without error (cerr != nil && err == nil). The GET for the instance's alternate ID otherwise succeeded; only connection teardown failed. Because OtherID uses a named return, this close error replaces the successful result with an error, which then surfaces wrapped as "retrieve other id failed" in Execute.

Source

Thrown at spinnaker/terminator.go:152

		log.Fatalf("chronos.jsonPayload could not marshal data into json: %v", err)
	}

	return result
}

// OtherID returns the alternate instance id of an instance, if it exists
// If there is no alternate instance id, it returns an empty string
// This is used by Titus, where we also report the uuid
func (s Spinnaker) OtherID(ins chaosmonkey.Instance) (otherID string, err error) {
	url := s.instanceURL(ins.AccountName(), ins.RegionName(), ins.ID())
	resp, err := s.client.Get(url)
	if err != nil {
		return "", errors.Wrap(err, fmt.Sprintf("get failed on %s", url))
	}

	defer func() {
		if cerr := resp.Body.Close(); cerr != nil && err == nil {
			err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body from %s", url))
		}
	}()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return "", errors.Wrap(err, fmt.Sprintf("body read failed at %s", url))
	}

	// Example of response body:
	/*
		{
			...
			"health": [
				{
					"type": "Titus",
					"healthClass": "platform",
					"state": "Up"
				},

View on GitHub (pinned to eaa28fb761)

Solutions

Check whether the underlying data was still correct; if so, this is a teardown artifact — drain the body before closing to reduce it.
Use io.Copy(io.Discard, resp.Body) before Close, or ensure full reads (the code already does ioutil.ReadAll, so check for early-return paths).
Tune http.Transport keepalive settings (IdleConnTimeout, MaxIdleConnsPerHost) to avoid stale connections.
Correlate errors.Cause with LB/Gate logs for connection resets.
If it persists and blocks terminations, consider ignoring close errors on GET responses whose body was fully read.

Example fix

// before
if cerr := resp.Body.Close(); cerr != nil && err == nil {
	err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body from %s", url))
}
// after
// body fully read above; treat close error as non-fatal
_ = resp.Body.Close()

Defensive patterns

Strategy: try-catch

Validate before calling

// body is fully read via ioutil.ReadAll before close; drain defensively on early-return paths
io.Copy(io.Discard, resp.Body)

Try / catch

otherID, err := spinnaker.OtherID(ins)
if err != nil {
	if strings.Contains(err.Error(), "failed to close response body") {
		log.Printf("non-fatal close error looking up %s: %v", ins.ID(), err)
		// data was fine; retry or proceed per policy
	}
	return errors.Wrap(err, "retrieve other id failed")
}

Prevention

Drain resp.Body fully before Close to keep pooled connections healthy
Tune http.Transport (IdleConnTimeout, MaxIdleConnsPerHost) to avoid stale keepalives
Treat close errors on fully-read GET bodies as non-fatal in caller policy
Correlate with LB logs if reset-on-close errors recur

When it happens

Trigger: GET to the Spinnaker instance endpoint succeeded and the body was read, but resp.Body.Close() returned an error — typically a reset/aborted keepalive connection during teardown.

Common situations: Server or LB closing keepalive connections aggressively; stale pooled connections in http.Transport; network device interrupting idle connections between request completion and close.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/dadaf6bd858679a7. Report an issue: GitHub.