{"record":{"id":"1733d83c66204a47","repo":"tinyhumansai/openhuman","slug":"workflow-run-run-id-vanished-mid-phase","errorCode":null,"errorMessage":"workflow run {run_id} vanished mid-phase","messagePattern":"workflow run (.+?) vanished mid-phase","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"src/openhuman/agent/orchestration/workflow_runs/engine.rs","lineNumber":515,"sourceCode":"/// cancellation landed mid-phase (the terminal status is persisted first).\n#[allow(clippy::too_many_arguments)]\npub(super) async fn execute_phase(\n    config: &Config,\n    run_id: &str,\n    definition: &WorkflowDefinition,\n    session: &crate::openhuman::agent::orchestration::AgentOrchestrationSession,\n    cancel: &Arc<AtomicBool>,\n    model_override: Option<String>,\n    phase: &WorkflowPhase,\n    total_spawned: u32,\n) -> Result<PhaseExecOutcome> {\n    use crate::openhuman::agent::orchestration::{\n        AgentStatus, SpawnAgentRequest, WaitAgentOptions,\n    };\n\n    // Reload so the phase state we mutate + persist is the latest projection.\n    let run = get_workflow_run(&config.workspace_dir, run_id)?\n        .ok_or_else(|| anyhow!(\"workflow run {run_id} vanished mid-phase\"))?;\n    let mut phase_states = run.phase_states.clone();\n    let mut child_run_ids = run.child_run_ids.clone();\n    // Children launched *this* phase (the reducer delta added to `total_spawned`).\n    let mut spawned_this_phase: u32 = 0;\n\n    log::debug!(\n        target: LOG_TARGET,\n        \"[workflow_run_engine] phase.start run={run_id} phase={} agents={} spawned_so_far={}\",\n        phase.name,\n        phase.agent_ids.len(),\n        total_spawned\n    );\n    set_phase_status(&mut phase_states, &phase.name, PHASE_RUNNING, None);\n    persist(\n        config,\n        &run,\n        phase_states.clone(),\n        child_run_ids.clone(),","sourceCodeStart":497,"sourceCodeEnd":533,"githubUrl":"https://github.com/tinyhumansai/openhuman/blob/a221052e0df5b1f7598fceba7329fd1af95d6699/src/openhuman/agent/orchestration/workflow_runs/engine.rs#L497-L533","documentation":"Thrown inside workflow_runs::engine phase execution when the run row — reloaded so the engine mutates the latest phase_states projection — is suddenly missing from the ledger. Same failure class as 'vanished mid-loop', detected at phase start instead of between phases: the run was deleted by something outside the engine while it held a live run id.","triggerScenarios":"The run ledger row is deleted (workspace wipe, DB file removal, concurrent delete call) after phase selection succeeded but before/during execute_phase's reload. Long phases widen the window; any external ledger mutation during execution triggers it.","commonSituations":"Workspace cleanup jobs with no awareness of running workflows; running integration tests against a workspace a daemon also cleans; deleting runs from a UI to 'tidy up' while they execute.","solutions":["Make run deletion exclusive: refuse to delete runs whose status is Running, or stop the run first via cancel","Coordinate workspace resets with the engine (drain runs before resetting)","Recover by starting a new run with the same input once the interference is gone"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"match execute_phase(config, run_id, session, cancel, model, phase, spawned).await {\n    Err(e) if e.to_string().contains(\"vanished mid-phase\") => {\n        // external deletion during execution — abort the loop, report interference\n        abort_and_report_data_loss(run_id, e)\n    }\n    other => other,\n}","preventionTips":["Gate workspace resets/cleanups on quiescence: no Running workflow runs in the ledger","Cancel a run through its cancel flag (engine persists Interrupted cleanly) instead of deleting its row","Keep exactly one process writing the run ledger per workspace"],"tags":["workflow-runs","concurrency","data-loss","internal-invariant"],"backgroundTag":null,"analyzedSha":"a221052e0df5b1f7598fceba7329fd1af95d6699","analyzedAt":"2026-08-16T12:47:06.542Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}