Netflix/chaosmonkey · error

POST to %s failed, (body '%s')

Error message

POST to %s failed, (body '%s')

What it means

Execute throws this when the http.Client.Post to the Spinnaker task endpoint fails at the transport level — no HTTP response was received at all. The message embeds the URL and the exact kill JSON payload that was attempted, so you can see precisely what request failed. This is the step that actually issues the instance kill; if you see it, no termination happened.

Source

Thrown at spinnaker/terminator.go:84

// Kill implements term.Terminator.Kill
func (t fakeTerminator) Execute(trm chaosmonkey.Termination) error {
	return nil
}

// Execute implements term.Terminator.Execute
func (s Spinnaker) Execute(trm chaosmonkey.Termination) (err error) {
	ins := trm.Instance
	url := s.tasksURL(ins.AppName())

	otherID, err := s.OtherID(ins)
	if err != nil {
		return errors.Wrap(err, "retrieve other id failed")
	}

	payload := killJSONPayload(ins, otherID, s.user)
	resp, err := s.client.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		return errors.Wrap(err, fmt.Sprintf("POST to %s failed, (body '%s')", url, string(payload)))
	}

	defer func() {
		if cerr := resp.Body.Close(); cerr != nil && err == nil {
			err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body of %s", url))
		}
	}()

	if resp.StatusCode != http.StatusOK {
		log.Printf("Unexpected response: %d", resp.StatusCode)
		contents, err := ioutil.ReadAll(resp.Body)
		if err != nil {
			return errors.Wrap(err, "failed to read response body")
		}
		return fmt.Errorf("unexpected response code: %d, body: %s", resp.StatusCode, string(contents))
	}

	return nil

View on GitHub (pinned to eaa28fb761)

Solutions

1. Check Spinnaker Gate is reachable: curl the URL in the error message from the host running chaosmonkey.
2. Verify the spinnaker endpoint configuration (scheme, host, port, path) in chaosmonkey config/env.
3. Check the embedded payload in the error to confirm the kill request body is well-formed and targets the right instance.
4. Inspect Gate logs and network paths (LB, security groups, DNS) for connection failures.
5. Add retry with backoff for transient transport errors, and ensure the http.Client timeout is appropriate for task submission.

u, err := url.Parse(spinnakerEndpoint)
if err != nil || u.Scheme == "" || u.Host == "" {
	return fmt.Errorf("invalid spinnaker endpoint: %q", spinnakerEndpoint)
}
// connectivity pre-check
resp, err := client.Get(spinnakerEndpoint + "/health")
if err != nil {
	return fmt.Errorf("spinnaker gate unreachable: %v", err)
}
resp.Body.Close()

resp, err := s.client.Post(url, "application/json", bytes.NewReader(payload))
if err != nil {
	if isRetryableNetErr(errors.Cause(err)) {
		// retry with backoff
	}
	return errors.Wrap(err, fmt.Sprintf("POST to %s failed, (body '%s')", url, string(payload)))
}

When it happens

Trigger: POST <spinnaker-api>/tasks with the killJSONPayload body fails: DNS failure, connection refused/reset, TLS error, or client timeout while talking to Spinnaker Gate.

Common situations: Spinnaker Gate is down or unreachable from where chaosmonkey runs; incorrect SPINNAKER endpoint URL in config; firewall/security-group rules blocking egress; TLS certificate mismatch; Gate overloaded and dropping connections during a large experiment.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• http get failed at %s
• http get failed at %s
• get failed on %s
• body read failed at %s
• could not retrieve list of apps from spinnaker url %s: %v

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/f57e2c20c6687ac2. Report an issue: GitHub.