{"record":{"id":"d89b720871c4e390","repo":"zeroclaw-labs/zeroclaw","slug":"terminal-persistence-failed-for-run-active-run","errorCode":null,"errorMessage":"terminal persistence failed for run {}; active run and admission claim remain retained: {}","messagePattern":"terminal persistence failed for run (.+?); active run and admission claim remain retained: (.+?)","errorType":"exception","errorClass":"TerminalPersistenceRetained","httpStatus":null,"severity":"error","filePath":"crates/zeroclaw-runtime/src/sop/engine.rs","lineNumber":1183,"sourceCode":"    }\n\n    /// Persist a run that has reached a terminal state and release its claim atomically.\n    fn persist_terminal(&self, run: &SopRun) -> Result<()> {\n        let mut pr = PersistedRun::new(run.clone(), now_iso8601(), run.trigger_event.source);\n        // The terminal write is the run's final revision; advance past the last\n        // active snapshot so the store's revision guard accepts it.\n        pr.revision = self.next_run_revision(&run.run_id);\n        self.store.finish_run(&run.run_id, &pr).map_err(|e| {\n            ::zeroclaw_log::record!(\n                WARN,\n                ::zeroclaw_log::Event::new(module_path!(), ::zeroclaw_log::Action::Reject)\n                    .with_outcome(::zeroclaw_log::EventOutcome::Failure)\n                    .with_attrs(\n                        ::serde_json::json!({\"run_id\": run.run_id, \"error\": e.to_string()})\n                    ),\n                \"SOP engine: terminal persistence failed; run and admission claim remain active\"\n            );\n            anyhow::Error::new(TerminalPersistenceRetained {\n                run_id: run.run_id.clone(),\n                source: e,\n            })\n        })?;\n        self.notify_run(run, false);\n        Ok(())\n    }\n\n    /// Terminal counterpart to `persist_active_with_gate_event`: persist the\n    /// terminal run, release its claim, and append the gate-resolution ledger row\n    /// in one store transaction.\n    fn persist_terminal_with_gate_event(&self, run: &SopRun, event: &SopEventRecord) -> Result<()> {\n        let mut pr = PersistedRun::new(run.clone(), now_iso8601(), run.trigger_event.source);\n        pr.revision = self.next_run_revision(&run.run_id);\n        self.store\n            .finish_run_with_event(&run.run_id, &pr, event)\n            .map_err(|e| {\n                ::zeroclaw_log::record!(","sourceCodeStart":1165,"sourceCodeEnd":1201,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-runtime/src/sop/engine.rs#L1165-L1201","documentation":"persist_terminal writes a run's terminal state via store.finish_run (with a pre-computed next revision so the store's revision guard accepts it) and releases its admission claim atomically; when the store write fails, the engine wraps the StoreError in TerminalPersistenceRetained (engine.rs:127-146) and returns it, deliberately keeping the run in active_runs with its claim retained so the terminal decision can be retried instead of leaving a half-finished run. The Display text names the run_id and chains the underlying store fault via source(). It is the fail-closed counterpart to dropping state: on failure, nothing is released.","triggerScenarios":"A run reaching a terminal Success/Fail state while store.finish_run errors — store I/O failure, a locked or corrupt store file, disk full, or a revision conflict on the terminal write.","commonSituations":"SQLite/store file on a full or read-only volume; store lock held by a crashed process; schema migration applied under a running engine; NFS/store path latency causing write failures.","solutions":["Read the chained source error (the {source} at the end of the message) — it names the actual store fault to fix","Restore store health (disk space, permissions, lock cleanup), then re-resolve the run: the retained active run and admission claim make the retry safe by design","Do not force-remove the run from active_runs to \"clean up\" — that orphans the admission claim and leaks a slot","If it recurs, capture the WARN log line \"terminal persistence failed\" with its run_id and error attrs for maintainers"],"exampleFix":"// before\nengine.resolve_gate(request).await?; // TerminalPersistenceRetained aborts the handler outright\n\n// after — store recovered? the run is still retained, so retry the terminal resolution\nlet mut last = None;\nfor attempt in 0..3 {\n    match engine.resolve_gate(request.clone()).await {\n        Ok(outcome) => { last = None; break; }\n        Err(e) if e.to_string().contains(\"terminal persistence failed\") => {\n            backoff(attempt).await; // store unhealthy; give it time to recover\n            last = Some(e);\n        }\n        Err(e) => return Err(e),\n    }\n}\nif let Some(e) = last { return Err(e); }","handlingStrategy":"retry","validationCode":null,"typeGuard":"fn is_terminal_persistence_retained(e: &anyhow::Error) -> bool {\n    // struct is private today; match the stable Display prefix or ask upstream for a classifier\n    e.to_string().starts_with(\"terminal persistence failed for run \")\n}","tryCatchPattern":"match engine.resolve_gate(request).await {\n    Ok(outcome) => outcome,\n    Err(e) if is_terminal_persistence_retained(&e) => {\n        // run and claim are retained: fix store health, then re-resolve\n        alert_operator(&e); schedule_retry();\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Monitor store health (disk space, lock contention) where the SOP store file lives","Never manually delete active runs to clear this state — the admission claim leaks","Capture the WARN \"terminal persistence failed\" log line with run_id for postmortems","Keep the store on a local, writable filesystem rather than flaky network mounts"],"tags":["rust","sop","persistence","state-store","durability","fail-closed"],"backgroundTag":"database-write-failed","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}