zeroclaw-labs/zeroclaw · warning · ResumeAtCapacity

run {} ({}) cannot resume yet: execution slots are full; it

Error message

run {} ({}) cannot resume yet: execution slots are full; it stays parked and re-resolvable once a slot frees

What it means

When a parked run is resumed (approval or deterministic checkpoint), reacquire_claim_on_resume re-admits it through store.try_claim_run under the SOP's per-SOP max_concurrent and the engine's global max_concurrent_total; when admission returns Ok(None) the engine returns the typed ResumeAtCapacity marker (engine.rs:156-171). The struct's own doc is explicit that this is routine BACKPRESSURE, not a fault: the run stays parked and re-resolvable, resolve_gate reports DeferredAtCapacity, and a later approval attempt or the timeout tick's retry resumes it once a slot frees. The public helper err_is_resume_at_capacity(err) exists so callers (e.g. the gateway resume endpoint) can render it as HTTP 503 instead of logging a failure.

Source

Thrown at crates/zeroclaw-runtime/src/sop/engine.rs:1112

        // pre-flights (`can_clear_waiting_gate` / `can_advance_deterministic_step`)
        // already proved the SOP is still loaded before we reach here; if it somehow
        // is not, fail closed rather than resume uncounted.
        let per_sop_cap = self
            .get_sop(&sop_name)
            .map(|sop| sop.max_concurrent as usize);
        let Some(per_sop_cap) = per_sop_cap else {
            return Err(anyhow::Error::msg(format!(
                "failed to re-acquire exec claim on resume for run {rid}: SOP '{sop_name}' no longer loaded"
            )));
        };
        match self.store.try_claim_run(
            &rid,
            &sop_name,
            per_sop_cap,
            self.config.max_concurrent_total,
        ) {
            Ok(Some(_token)) => Ok(()),
            Ok(None) => Err(anyhow::Error::new(ResumeAtCapacity {
                run_id: rid,
                sop_name,
            })),
            Err(e) => {
                ::zeroclaw_log::record!(
                    WARN,
                    ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Note)
                        .with_outcome(::zeroclaw_log::EventOutcome::Failure)
                        .with_attrs(::serde_json::json!({
                            "run_id": rid.as_str(),
                            "error": e.to_string(),
                        })),
                    "SOP engine: resume aborted, could not re-acquire the run admission claim (fail-closed)"
                );
                Err(anyhow::Error::msg(format!(
                    "failed to re-acquire exec claim on resume for run {rid}: {e}"
                )))
            }

View on GitHub (pinned to 88bb9c8533)

Solutions

1. Retry the same resolve/resume call after a slot frees — the engine's timeout tick also retries automatically, and nothing about the run was lost or advanced
2. If the burst is expected behavior, raise the SOP definition's max_concurrent or the engine config's max_concurrent_total
3. Surface it as backpressure (HTTP 503, "retry later") via err_is_resume_at_capacity instead of a 500-style fault
4. If capacity never frees, look for stuck executing runs holding claims (lease reaper / reap output) rather than re-approving harder

Example fix

// before
let outcome = engine.resolve_gate(request).await?; // capacity error bubbles as an opaque failure

// after
match engine.resolve_gate(request).await {
    Ok(outcome) => outcome,
    Err(e) if sop::engine::err_is_resume_at_capacity(&e) => {
        return (StatusCode::SERVICE_UNAVAILABLE, "execution slots full; retry later").into_response();
    }
    Err(e) => return handle_fault(e),
}

// the library ships this classifier; use it instead of string matching
fn at_capacity(e: &anyhow::Error) -> bool {
    zeroclaw_runtime::sop::engine::err_is_resume_at_capacity(e)
}

match engine.resolve_gate(request).await {
    Ok(outcome) => outcome,
    Err(e) if err_is_resume_at_capacity(&e) => {
        // backpressure: 503 + Retry-After, or scheduled retry; run stays parked
    }
    Err(e) => return Err(e), // real fault
}

When it happens

Trigger: Approving or resuming a run parked at a HITL gate / deterministic checkpoint while executing runs already saturate that SOP's max_concurrent or the engine's max_concurrent_total — the classic case is a burst of runs that all parked (releasing their slots) and then get approved simultaneously.

Common situations: Bulk-approving a backlog of parked runs at once; max_concurrent_total sized below routine simultaneous approvals; slow terminal steps holding slots while operators click through a queue; load tests that fan out approvals.

Related errors

• amqp.{}: dispatch = {:?} routes to the SOP engine but no SOP
• exceeded {} reconnect attempts, giving up
• cannot record checkpoint approval vote for run {run_id}: its
• cannot record approval vote for run {run_id}: its parked sna
• Cannot start SOP '{}': cooldown or concurrency limit reached

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/529d5c7df3761f18. Report an issue: GitHub.