What causes "unexpected status code: %d. error: %s" in Netflix/chaosmonkey?

Read the embedded Spinnaker error string to identify the server-side cause

unexpected status code: %d. error: %s

Error message

unexpected status code: %d. error: %s

What it means

Variant of the non-200 handling in OtherID(): when Spinnaker responds with a non-200 status and the decoded body includes a populated Error field, that server-side error string is surfaced instead of the raw body.

Source

Thrown at spinnaker/terminator.go:195

		}
	*/

	var fields struct {
		Health []map[string]interface{} `json:"health"`
		Error  string                   `json:"error"`
	}

	err = json.Unmarshal(body, &fields)
	if err != nil {
		return "", errors.Wrap(err, fmt.Sprintf("json unmarshal failed, body: %s", body))
	}

	if resp.StatusCode != http.StatusOK {
		if fields.Error == "" {
			return "", fmt.Errorf("unexpected status code: %d. body: %s", resp.StatusCode, body)
		}

		return "", fmt.Errorf("unexpected status code: %d. error: %s", resp.StatusCode, fields.Error)
	}

	// In some cases, an instance may be missing health information.
	// We just return a blank otherID in that case
	if len(fields.Health) < 2 {
		return "", nil
	}

	otherID, ok := fields.Health[1]["instanceId"].(string)
	if !ok {
		return "", nil
	}

	// If the instance id is the same, there is no alternate
	if ins.ID() == otherID {
		return "", nil
	}

View on GitHub (pinned to eaa28fb761)

Solutions

Read the embedded Spinnaker error string to identify the server-side cause
Validate the instance ID and cloud account before terminating
Treat instance-not-found as idempotent success where safe
Check Spinnaker/cloud provider health for backend errors

Example fix

// before
return "", fmt.Errorf("unexpected status code: %d. error: %s", resp.StatusCode, fields.Error)
// after
if resp.StatusCode == http.StatusNotFound {
    log.Printf("instance already gone: %s", fields.Error)
    return "", nil
}
return "", fmt.Errorf("unexpected status code: %d. error: %s", resp.StatusCode, fields.Error)

Defensive patterns

Strategy: try-catch

Validate before calling

// confirm the target instance ID exists in Spinnaker before executing

Try / catch

if err != nil {
    // error embeds Spinnaker's server-side error string
    log.Printf("spinnaker rejected request: %v", err)
    if strings.Contains(err.Error(), "not found") {
        return nil // idempotent
    }
    return err
}

Prevention

Parse the embedded error field, don't just retry blindly
Validate instance/account before calling
Handle concurrent terminations gracefully
Monitor Spinnaker backend health

When it happens

Trigger: Calling OtherID when Spinnaker returns 4xx/5xx AND the JSON body's error field is non-empty, e.g. instance not found or backend cloud-provider error relayed by Spinnaker.

Common situations: Invalid instance ID, instance deleted concurrently, Spinnaker forwarding a cloud API failure.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/0f286bd68523120e. Report an issue: GitHub.