{"record":{"id":"f653258421fbb21d","repo":"tinyhumansai/openhuman","slug":"workflow-run-run-id-vanished-mid-loop","errorCode":null,"errorMessage":"workflow run {run_id} vanished mid-loop","messagePattern":"workflow run (.+?) vanished mid-loop","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"src/openhuman/agent/orchestration/workflow_runs/engine.rs","lineNumber":428,"sourceCode":"    Continue { spawned: u32 },\n    /// The run reached a terminal status (already persisted) — route to `done`.\n    Terminated,\n}\n\n/// `dispatch` step: reload the run, honour cancellation, and pick the next\n/// runnable phase (pending, all deps `completed`). When none remains, persist the\n/// terminal status (Completed / Failed) and return [`PhaseSelection::Terminated`].\npub(super) async fn select_next_phase(\n    config: &Config,\n    run_id: &str,\n    definition: &WorkflowDefinition,\n    cancel: &Arc<AtomicBool>,\n    session: &crate::openhuman::agent::orchestration::AgentOrchestrationSession,\n) -> Result<PhaseSelection> {\n    // Reload so we read the latest phase_states (and a resume picks up persisted\n    // progress).\n    let run = get_workflow_run(&config.workspace_dir, run_id)?\n        .ok_or_else(|| anyhow!(\"workflow run {run_id} vanished mid-loop\"))?;\n    let phase_states = run.phase_states.clone();\n    let child_run_ids = run.child_run_ids.clone();\n\n    // Cancellation check between phases.\n    if cancel.load(Ordering::SeqCst) {\n        log::debug!(\n            target: LOG_TARGET,\n            \"[workflow_run_engine] loop.cancelled run={run_id}\"\n        );\n        session.abort_all().await;\n        persist(\n            config,\n            &run,\n            phase_states,\n            child_run_ids,\n            WorkflowRunStatus::Interrupted,\n            None,\n            false,","sourceCodeStart":410,"sourceCodeEnd":446,"githubUrl":"https://github.com/tinyhumansai/openhuman/blob/a221052e0df5b1f7598fceba7329fd1af95d6699/src/openhuman/agent/orchestration/workflow_runs/engine.rs#L410-L446","documentation":"Thrown inside workflow_runs::engine::select_next_phase when the engine loop reloads the run row between phases and get_workflow_run returns None. The run existed when the loop started (its id came from start/resume) but its ledger row was deleted mid-execution — external interference with the run ledger, not a caller-input problem.","triggerScenarios":"A workflow run is executing while something deletes its row: workspace reset/wipe, ledger DB deleted or restored, or a concurrent delete-RPC/tool acting on the same run. The engine aborts at the next phase boundary with this error.","commonSituations":"Test harness resetting the workspace while a long workflow runs; two operators (or a script plus UI) managing the same workspace; backup-restore of the ledger underneath a live run.","solutions":["Stop deleting/resetting the workspace while runs are in flight — gate resets on no active runs","If the deletion was accidental, restart the workflow as a new run (the old one is unrecoverable)","Audit for concurrent processes writing the same run-ledger DB"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// in the engine loop caller — the row can vanish at any await point, so catch and classify\nlet result = run_engine_loop(config, run_id).await;\nmatch result {\n    Err(e) if e.to_string().contains(\"vanished mid-loop\") => {\n        log::error!(\"run {run_id} deleted mid-execution by an external writer\");\n        // surface as data-loss/interference, not a workflow failure; do not blind-retry\n        Err(e)\n    }\n    other => other,\n}","preventionTips":["Serialize ledger access: never reset/delete the workspace DB while runs are Running","Expose run deletion only through an API that refuses Running runs","Watch for 'vanished mid-loop/mid-phase' in logs as the canary for concurrent ledger writers"],"tags":["workflow-runs","concurrency","data-loss","internal-invariant"],"backgroundTag":null,"analyzedSha":"a221052e0df5b1f7598fceba7329fd1af95d6699","analyzedAt":"2026-08-16T12:47:06.542Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}