What causes "Cannot start SOP '{}': cooldown or concurrency limit reached" in zeroclaw-labs/zeroclaw?

Wait for in-flight runs of the SOP to finish (or its cooldown to elapse) and retry the start.

zeroclaw-labs/zeroclaw · error · anyhow::Error

Cannot start SOP '{}': cooldown or concurrency limit reached

Error message

Cannot start SOP '{}': cooldown or concurrency limit reached

What it means

claim_admission() (reached via reserve_run_slot) asks the admission ledger for a start token and got None: the SOP is in cooldown, its per-SOP max_concurrent is exhausted, or the global max_concurrent_total is reached. The SOP start is refused — this is backpressure, not corruption.

Source

Thrown at crates/zeroclaw-runtime/src/sop/engine.rs:929

    pub fn is_gate_reference_superseded(&self, run_id: &str, reference_revision: u32) -> bool {
        self.active_runs.get(run_id).is_some_and(|run| {
            run.revision != reference_revision && !self.is_park_persist_pending(run_id)
        })
    }

    /// Admit a run through the store CAS claim before it becomes locally active.
    /// The durable store is the concurrency source of truth; `active_runs` is the
    /// execution cache/status surface.
    fn claim_admission(&self, run_id: &str, sop: &Sop) -> Result<ClaimToken> {
        match self.store.try_claim_run(
            run_id,
            &sop.name,
            sop.max_concurrent as usize,
            self.config.max_concurrent_total,
        ) {
            Ok(Some(token)) => Ok(token),
            Ok(None) => {
                bail!(
                    "Cannot start SOP '{}': cooldown or concurrency limit reached",
                    sop.name
                );
            }
            Err(e) => Err(anyhow::Error::new(e)),
        }
    }

    fn release_claim_best_effort(&self, token: &ClaimToken) {
        if let Err(e) = self.store.release_claim(token) {
            ::zeroclaw_log::record!(
                WARN,
                ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Note)
                    .with_outcome(::zeroclaw_log::EventOutcome::Unknown)
                    .with_attrs(::serde_json::json!({
                        "run_id": token.run_id.as_str(),
                        "error": e.to_string(),
                    })),

View on GitHub (pinned to 88bb9c8533)

Solutions

Wait for in-flight runs of the SOP to finish (or its cooldown to elapse) and retry the start.
If the load is legitimate, raise the SOP's max_concurrent and/or the global max_concurrent_total in config.
Check for stuck/long-running runs holding slots — resolve or time them out.
For bursty sources, prefer the dispatch path with an admission policy (Coalesce/Defer) instead of direct starts.

Example fix

# before
max_concurrent = 1
# burst of 3 triggers -> 2nd and 3rd starts bail

# after
max_concurrent = 3
# (or route triggers through dispatch with coalesce/defer policy)

Defensive patterns

Strategy: retry

Validate before calling

match engine.evaluate_admission(sop_name) {
    SopAdmission::Admit => engine.reserve_run_slot(sop_name),
    other => handle_non_admit(other), // wait, coalesce, or surface reason — do not call reserve
}

Try / catch

let mut delay = Duration::from_millis(500);
loop {
    match engine.start_sop(sop_name, input.clone()) {
        Ok(run) => break run,
        Err(e) if e.to_string().contains("cooldown or concurrency limit reached") && delay <= Duration::from_secs(30) => {
            tokio::time::sleep(delay).await;
            delay *= 2;
        }
        Err(e) => return Err(e),
    }
}

Prevention

Size max_concurrent and max_concurrent_total to your real trigger load; measure before tuning down.
Watch for stuck runs holding slots — alert on long-executing SOP runs.
Prefer the dispatch API with admission policies for bursty triggers instead of direct start calls.

When it happens

Trigger: Starting a SOP (sop_execute / start paths that reserve a run slot) while: the SOP's cooldown window after a prior run is active, exec counts for the SOP are at max_concurrent, or total running executions are at config.max_concurrent_total.

Common situations: Bursty triggers (webhooks, cron fan-out) exceeding configured concurrency; cooldown set to debounce repeated triggers and a legitimate replay arriving inside the window; max_concurrent_total tuned too low for the workload; stuck runs holding slots.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/3ece2cd93ea6ce56. Report an issue: GitHub.