Netflix/chaosmonkey · error

retrieve other id failed

Error message

retrieve other id failed

What it means

Execute wraps any failure from s.OtherID(ins) with the message "retrieve other id failed". OtherID queries the Spinnaker instance endpoint to find an instance's alternate ID (the Titus uuid), so this error means chaosmonkey could not look up the alternate instance id and the termination (kill) request was never sent. The underlying cause (HTTP failure, body read error, unmarshal error, or non-200 status from OtherID) is chained via errors.Wrap.

Source

Thrown at spinnaker/terminator.go:78

// tasksURL returns the Spinnaker tasks URL associated with an app
func (s Spinnaker) tasksURL(appName string) string {
	return s.appURL(appName) + "/tasks"
}

// Kill implements term.Terminator.Kill
func (t fakeTerminator) Execute(trm chaosmonkey.Termination) error {
	return nil
}

// Execute implements term.Terminator.Execute
func (s Spinnaker) Execute(trm chaosmonkey.Termination) (err error) {
	ins := trm.Instance
	url := s.tasksURL(ins.AppName())

	otherID, err := s.OtherID(ins)
	if err != nil {
		return errors.Wrap(err, "retrieve other id failed")
	}

	payload := killJSONPayload(ins, otherID, s.user)
	resp, err := s.client.Post(url, "application/json", bytes.NewReader(payload))
	if err != nil {
		return errors.Wrap(err, fmt.Sprintf("POST to %s failed, (body '%s')", url, string(payload)))
	}

	defer func() {
		if cerr := resp.Body.Close(); cerr != nil && err == nil {
			err = errors.Wrap(cerr, fmt.Sprintf("failed to close response body of %s", url))
		}
	}()

	if resp.StatusCode != http.StatusOK {
		log.Printf("Unexpected response: %d", resp.StatusCode)
		contents, err := ioutil.ReadAll(resp.Body)
		if err != nil {

View on GitHub (pinned to eaa28fb761)

Solutions

  1. Look at errors.Cause / the wrapped error to see which stage of OtherID failed (get, body read, unmarshal, or non-200 status).
  2. Verify the instance still exists in Spinnaker before terminating; a 404 means it already went away — treat as already-terminated.
  3. Check Spinnaker Gate availability and logs at the failure time.
  4. Validate that the app/account/region/instance ID stored by chaosmonkey matches the live Spinnaker state.
  5. Re-run the experiment step; transient Gate errors often resolve on retry.
Defensive patterns

Strategy: try-catch

Validate before calling

// before executing the termination, confirm the instance still exists
resp, err := client.Get(instanceURL(app, account, region, instanceID))
if err != nil || resp.StatusCode == http.StatusNotFound {
	return fmt.Errorf("instance %s/%s/%s no longer exists; skipping termination", app, region, instanceID)
}

Try / catch

otherID, err := spinnaker.OtherID(ins)
if err != nil {
	log.Printf("terminate %s: retrieve other id failed: %v (cause: %v)", ins.ID(), err, errors.Cause(err))
	if isNotFound(err) {
		return nil // already terminated; treat as success
	}
	return err
}

Prevention

When it happens

Trigger: chaosmonkey.Termination execution calls Spinnaker.Execute, which calls OtherID for the target instance; the GET <api>/<account>/<region>/<instanceId> request fails at any stage (transport error, body read error, JSON parse error, or non-200 status).

Common situations: The instance was already terminated before Execute ran, so Spinnaker returns 404; Titus instances whose task data changed or whose uuid lookup fails; temporary Spinnaker Gate outage during the termination step of the chaos experiment; wrong account/region/instance ID data in the chaosmonkey datastore.

Related errors

AI-assisted analysis of Netflix/chaosmonkey@eaa28fb761 (2026-09-03). Data as JSON: /api/errors/0e17df161dcf5c8e. Report an issue: GitHub.