{"record":{"id":"81feb0bd9370f04d","repo":"Hmbown/CodeWhale","slug":"dynamic-tool-settlement-task-ended-before-acknowle","errorCode":null,"errorMessage":"Dynamic tool settlement task ended before acknowledgement","messagePattern":"Dynamic tool settlement task ended before acknowledgement","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/tui/src/runtime_threads.rs","lineNumber":4328,"sourceCode":"                    }\n                    Err(format!(\n                        \"Dynamic tool settlement task panicked: {}\",\n                        panic_payload_message(&*payload)\n                    ))\n                }\n            };\n            let _ = ack_tx.send(result);\n        });\n        ack_rx\n    }\n\n    async fn await_dynamic_tool_settlement(\n        ack: oneshot::Receiver<std::result::Result<DynamicToolSettlementAck, String>>,\n    ) -> Result<DynamicToolSettlementAck> {\n        match ack.await {\n            Ok(Ok(ack)) => Ok(ack),\n            Ok(Err(error)) => bail!(\"{error}\"),\n            Err(_) => bail!(\"Dynamic tool settlement task ended before acknowledgement\"),\n        }\n    }\n\n    async fn settle_dynamic_tool_timeout(\n        &self,\n        claim: ClaimedDynamicToolSettlement,\n        timeout: Duration,\n    ) -> Result<()> {\n        let ack = self\n            .spawn_dynamic_tool_settlement(claim, DynamicToolTerminalOutcome::Timeout { timeout });\n        Self::await_dynamic_tool_settlement(ack).await?;\n        Ok(())\n    }\n\n    async fn settle_dynamic_tools_for_terminal_turn(\n        &self,\n        thread_id: &str,\n        turn_id: &str,","sourceCodeStart":4310,"sourceCodeEnd":4346,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/0c42157ee52f9d55af2b506d71b46249910f77d3/crates/tui/src/runtime_threads.rs#L4310-L4346","documentation":"await_dynamic_tool_settlement's oneshot ack receiver returned Err - the settlement task's future ended without ever calling ack_tx.send. Every normal and panic path sends an ack, so a dropped channel means the task was aborted before completion (runtime shutdown dropping the tokio task) or the future was cancelled between claim and ack.","triggerScenarios":"The spawned settlement task (spawn_dynamic_tool_settlement) is dropped because the runtime is shutting down or the tokio runtime is torn down while a settlement is in flight; distinct from 635, which covers acknowledged failures. See runtime_threads.rs:4328.","commonSituations":"Process exit or runtime shutdown racing result submission; a test harness dropping the runtime mid-settlement; abrupt cancellation of the task's parent.","solutions":["Check whether the process/runtime is shutting down - if so, the call's durable state is unknown and storage must be inspected before any retry","Ensure submissions are drained before dropping the Runtime manager (quiesce on shutdown)","If it happens outside shutdown, inspect the thread's JSONL log for the terminal event to learn whether the append landed","Do not blindly retry: if the append did land, a retry can duplicate a terminal receipt - the indeterminate guard will catch it, so reconcile storage first"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Missing ack: check for shutdown first, then inspect storage.\nmatch manager.submit_dynamic_tool_result(thread_id, turn_id, call_id, result).await {\n    Ok(accepted) => Ok(accepted),\n    Err(e) if e.to_string().contains(\"ended before acknowledgement\") => {\n        if shutting_down() { return Err(e); } // quiesce path: outcome unknown\n        tracing::error!(%call_id, \"settlement task dropped; inspect JSONL for terminal event\");\n        Err(e)\n    }\n    Err(e) => Err(e),\n}","preventionTips":["Drain in-flight settlements (quiesce) before dropping the Runtime manager","Avoid tearing down the tokio runtime while submissions are outstanding","On missing ack, inspect the JSONL log for the terminal line before retrying - a landed append plus retry duplicates a receipt"],"tags":["dynamic-tools","shutdown","cancellation","acknowledgment","runtime"],"backgroundTag":"task-ended-before-acknowledgment","analyzedSha":"0c42157ee52f9d55af2b506d71b46249910f77d3","analyzedAt":"2026-08-20T21:50:45.477Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}