JuliusBrussee/caveman · error

experiment is stopped

Error message

experiment %q is stopped

What it means

SwitchExperimentArm refuses to change arms on an experiment whose stopped_at timestamp is already set. Once an experiment is stopped its interval history is final, so any further arm switches are rejected.

Solutions

1. Create a new experiment with StartExperiment instead of switching arms on the stopped one.
2. Check the experiment's StoppedAt field (via LoadExperiment/ListExperiments) before switching.
3. Pick a fresh unique label for the new experiment run.
4. If stopping was a mistake, stop using the record — this API has no un-stop; recreate the experiment.

Example fix

// before
exp, err := store.SwitchExperimentArm(label, "off")
// after
exp, err := store.LoadExperiment(label)
if err == nil && exp.StoppedAt != "" {
    return store.StartExperiment(label + "-2")
}
return store.SwitchExperimentArm(label, "off")

exp, err := store.LoadExperiment(label)
if err == nil && exp.StoppedAt != "" { // stopped; don't switch
    return ErrAlreadyStopped
}

_, err := store.SwitchExperimentArm(label, arm)
if err != nil && strings.Contains(err.Error(), "is stopped") {
    // start a new experiment instead
}

When it happens

Trigger: Calling Store.SwitchExperimentArm on a label that was previously passed to StopExperiment (experiments.stopped_at non-empty).

Common situations: A script that stops experiments at the end of a run then attempts to re-enable an arm; reusing a familiar experiment label after it was finalized; a cron job racing with a manual stop.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• arm must be or
• CAVEMAN_CCR_MAX_BYTES must be a positive integer
• ccr max storage bytes must be positive
• ccr: unknown lifecycle
• close open arm

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/33fdee6461f65b9c. Report an issue: GitHub.

Appendix: source

Thrown at proxy/internal/store/learn_experiment.go:125

		return Experiment{}, fmt.Errorf("open first arm: %w", err)
	}
	return s.LoadExperiment(label)
}

// SwitchExperimentArm closes the open arm and opens the other one. Switching to
// the arm already running is a no-op rather than an error: it keeps a repeated
// call from fragmenting the interval history.
func (s *Store) SwitchExperimentArm(label, arm string) (Experiment, error) {
	arm = strings.ToLower(strings.TrimSpace(arm))
	if arm != experimentArmOn && arm != experimentArmOff {
		return Experiment{}, fmt.Errorf("arm must be %q or %q", experimentArmOn, experimentArmOff)
	}
	experiment, err := s.LoadExperiment(label)
	if err != nil {
		return Experiment{}, err
	}
	if experiment.StoppedAt != "" {
		return Experiment{}, fmt.Errorf("experiment %q is stopped", experiment.Label)
	}
	for _, existing := range experiment.Arms {
		if existing.EndedAt == "" && existing.Arm == arm {
			return experiment, nil
		}
	}
	now := learnOutcomeClock().UTC().Format(time.RFC3339)
	if _, err := s.db.Exec(`UPDATE experiment_arms SET ended_at = ?
		WHERE experiment_id = ? AND ended_at IS NULL`, now, experiment.id); err != nil {
		return Experiment{}, fmt.Errorf("close open arm: %w", err)
	}
	if _, err := s.db.Exec(`INSERT INTO experiment_arms (experiment_id, arm, started_at)
		VALUES (?, ?, ?)`, experiment.id, arm, now); err != nil {
		return Experiment{}, fmt.Errorf("open arm %q: %w", arm, err)
	}
	return s.LoadExperiment(label)
}

View on GitHub (pinned to 3ee70a1026)