{"record":{"id":"529d5c7df3761f18","repo":"zeroclaw-labs/zeroclaw","slug":"run-cannot-resume-yet-execution-slots-are","errorCode":null,"errorMessage":"run {} ({}) cannot resume yet: execution slots are full; it stays parked and re-resolvable once a slot frees","messagePattern":"run (.+?) \\((.+?)\\) cannot resume yet: execution slots are full; it stays parked and re-resolvable once a slot frees","errorType":"exception","errorClass":"ResumeAtCapacity","httpStatus":503,"severity":"warning","filePath":"crates/zeroclaw-runtime/src/sop/engine.rs","lineNumber":1112,"sourceCode":"        // pre-flights (`can_clear_waiting_gate` / `can_advance_deterministic_step`)\n        // already proved the SOP is still loaded before we reach here; if it somehow\n        // is not, fail closed rather than resume uncounted.\n        let per_sop_cap = self\n            .get_sop(&sop_name)\n            .map(|sop| sop.max_concurrent as usize);\n        let Some(per_sop_cap) = per_sop_cap else {\n            return Err(anyhow::Error::msg(format!(\n                \"failed to re-acquire exec claim on resume for run {rid}: SOP '{sop_name}' no longer loaded\"\n            )));\n        };\n        match self.store.try_claim_run(\n            &rid,\n            &sop_name,\n            per_sop_cap,\n            self.config.max_concurrent_total,\n        ) {\n            Ok(Some(_token)) => Ok(()),\n            Ok(None) => Err(anyhow::Error::new(ResumeAtCapacity {\n                run_id: rid,\n                sop_name,\n            })),\n            Err(e) => {\n                ::zeroclaw_log::record!(\n                    WARN,\n                    ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Note)\n                        .with_outcome(::zeroclaw_log::EventOutcome::Failure)\n                        .with_attrs(::serde_json::json!({\n                            \"run_id\": rid.as_str(),\n                            \"error\": e.to_string(),\n                        })),\n                    \"SOP engine: resume aborted, could not re-acquire the run admission claim (fail-closed)\"\n                );\n                Err(anyhow::Error::msg(format!(\n                    \"failed to re-acquire exec claim on resume for run {rid}: {e}\"\n                )))\n            }","sourceCodeStart":1094,"sourceCodeEnd":1130,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-runtime/src/sop/engine.rs#L1094-L1130","documentation":"When a parked run is resumed (approval or deterministic checkpoint), reacquire_claim_on_resume re-admits it through store.try_claim_run under the SOP's per-SOP max_concurrent and the engine's global max_concurrent_total; when admission returns Ok(None) the engine returns the typed ResumeAtCapacity marker (engine.rs:156-171). The struct's own doc is explicit that this is routine BACKPRESSURE, not a fault: the run stays parked and re-resolvable, resolve_gate reports DeferredAtCapacity, and a later approval attempt or the timeout tick's retry resumes it once a slot frees. The public helper err_is_resume_at_capacity(err) exists so callers (e.g. the gateway resume endpoint) can render it as HTTP 503 instead of logging a failure.","triggerScenarios":"Approving or resuming a run parked at a HITL gate / deterministic checkpoint while executing runs already saturate that SOP's max_concurrent or the engine's max_concurrent_total — the classic case is a burst of runs that all parked (releasing their slots) and then get approved simultaneously.","commonSituations":"Bulk-approving a backlog of parked runs at once; max_concurrent_total sized below routine simultaneous approvals; slow terminal steps holding slots while operators click through a queue; load tests that fan out approvals.","solutions":["Retry the same resolve/resume call after a slot frees — the engine's timeout tick also retries automatically, and nothing about the run was lost or advanced","If the burst is expected behavior, raise the SOP definition's max_concurrent or the engine config's max_concurrent_total","Surface it as backpressure (HTTP 503, \"retry later\") via err_is_resume_at_capacity instead of a 500-style fault","If capacity never frees, look for stuck executing runs holding claims (lease reaper / reap output) rather than re-approving harder"],"exampleFix":"// before\nlet outcome = engine.resolve_gate(request).await?; // capacity error bubbles as an opaque failure\n\n// after\nmatch engine.resolve_gate(request).await {\n    Ok(outcome) => outcome,\n    Err(e) if sop::engine::err_is_resume_at_capacity(&e) => {\n        return (StatusCode::SERVICE_UNAVAILABLE, \"execution slots full; retry later\").into_response();\n    }\n    Err(e) => return handle_fault(e),\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":"// the library ships this classifier; use it instead of string matching\nfn at_capacity(e: &anyhow::Error) -> bool {\n    zeroclaw_runtime::sop::engine::err_is_resume_at_capacity(e)\n}","tryCatchPattern":"match engine.resolve_gate(request).await {\n    Ok(outcome) => outcome,\n    Err(e) if err_is_resume_at_capacity(&e) => {\n        // backpressure: 503 + Retry-After, or scheduled retry; run stays parked\n    }\n    Err(e) => return Err(e), // real fault\n}","preventionTips":["Size max_concurrent_total above your worst realistic simultaneous-approval burst","Map this marker to 503/backpressure at API edges; never log it as ERROR","Rely on the engine's timeout tick to resume parked runs; avoid hand-rolled tight retry loops","Watch for stuck executing runs (lease reaper output) when capacity never frees"],"tags":["rust","sop","workflow-engine","backpressure","concurrency-limits","retry"],"backgroundTag":"concurrency-limit-backpressure","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}