What causes "unexpected status code: %d. body: %s" in Netflix/chaosmonkey?

Check the body: 404 usually means the instance no longer exists — handle idempotently

unexpected status code: %d. body: %s

Error message

unexpected status code: %d. body: %s

What it means

OtherID() fetches instance details from Spinnaker to determine an alternate instance ID. When the HTTP status is not 200 and the decoded response contains no Error field, this error includes the status code and raw body.

Source

Thrown at spinnaker/terminator.go:192

					...
				}
			],
		}
	*/

	var fields struct {
		Health []map[string]interface{} `json:"health"`
		Error  string                   `json:"error"`
	}

	err = json.Unmarshal(body, &fields)
	if err != nil {
		return "", errors.Wrap(err, fmt.Sprintf("json unmarshal failed, body: %s", body))
	}

	if resp.StatusCode != http.StatusOK {
		if fields.Error == "" {
			return "", fmt.Errorf("unexpected status code: %d. body: %s", resp.StatusCode, body)
		}

		return "", fmt.Errorf("unexpected status code: %d. error: %s", resp.StatusCode, fields.Error)
	}

	// In some cases, an instance may be missing health information.
	// We just return a blank otherID in that case
	if len(fields.Health) < 2 {
		return "", nil
	}

	otherID, ok := fields.Health[1]["instanceId"].(string)
	if !ok {
		return "", nil
	}

	// If the instance id is the same, there is no alternate
	if ins.ID() == otherID {

View on GitHub (pinned to eaa28fb761)

Solutions

Check the body: 404 usually means the instance no longer exists — handle idempotently
Refresh Spinnaker credentials for 401/403
Verify the instance ID/account passed to OtherID
Retry on 5xx; investigate Spinnaker server logs

Example fix

// before
return "", fmt.Errorf("unexpected status code: %d. body: %s", resp.StatusCode, body)
// after
if resp.StatusCode == http.StatusNotFound {
    return "", nil // instance already gone; treat as terminated
}
return "", fmt.Errorf("unexpected status code: %d. body: %s", resp.StatusCode, body)

Defensive patterns

Strategy: try-catch

Validate before calling

// verify the instance exists before calling OtherID/Execute
// list instances for the account and confirm the ID is present

Try / catch

otherID, err := terminator.Execute(...)
if err != nil {
    if strings.Contains(err.Error(), "404") {
        log.Printf("instance already gone; treating as terminated")
    } else {
        return err
    }
}

Prevention

Treat 404 as idempotent success for termination flows
Refresh credentials on 401/403
Validate instance IDs against the cloud provider
Retry 5xx with backoff

When it happens

Trigger: Calling OtherID (from Execute) when the instance-details request returns e.g. 404 (instance gone), 401/403 (auth), or 5xx and the JSON body has an empty Error field.

Common situations: Terminating an instance that was already terminated (404), stale credentials, Spinnaker outage with a non-JSON error page.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/4a060e9022310a65. Report an issue: GitHub.