Hmbown/CodeWhale · error · anyhow::Error

Dynamic tool settlement task ended before acknowledgement

Error message

Dynamic tool settlement task ended before acknowledgement

What it means

await_dynamic_tool_settlement's oneshot ack receiver returned Err - the settlement task's future ended without ever calling ack_tx.send. Every normal and panic path sends an ack, so a dropped channel means the task was aborted before completion (runtime shutdown dropping the tokio task) or the future was cancelled between claim and ack.

Source

Thrown at crates/tui/src/runtime_threads.rs:4328

                    }
                    Err(format!(
                        "Dynamic tool settlement task panicked: {}",
                        panic_payload_message(&*payload)
                    ))
                }
            };
            let _ = ack_tx.send(result);
        });
        ack_rx
    }

    async fn await_dynamic_tool_settlement(
        ack: oneshot::Receiver<std::result::Result<DynamicToolSettlementAck, String>>,
    ) -> Result<DynamicToolSettlementAck> {
        match ack.await {
            Ok(Ok(ack)) => Ok(ack),
            Ok(Err(error)) => bail!("{error}"),
            Err(_) => bail!("Dynamic tool settlement task ended before acknowledgement"),
        }
    }

    async fn settle_dynamic_tool_timeout(
        &self,
        claim: ClaimedDynamicToolSettlement,
        timeout: Duration,
    ) -> Result<()> {
        let ack = self
            .spawn_dynamic_tool_settlement(claim, DynamicToolTerminalOutcome::Timeout { timeout });
        Self::await_dynamic_tool_settlement(ack).await?;
        Ok(())
    }

    async fn settle_dynamic_tools_for_terminal_turn(
        &self,
        thread_id: &str,
        turn_id: &str,

View on GitHub (pinned to 0c42157ee5)

Solutions

Check whether the process/runtime is shutting down - if so, the call's durable state is unknown and storage must be inspected before any retry
Ensure submissions are drained before dropping the Runtime manager (quiesce on shutdown)
If it happens outside shutdown, inspect the thread's JSONL log for the terminal event to learn whether the append landed
Do not blindly retry: if the append did land, a retry can duplicate a terminal receipt - the indeterminate guard will catch it, so reconcile storage first

Defensive patterns

Strategy: try-catch

Try / catch

// Missing ack: check for shutdown first, then inspect storage.
match manager.submit_dynamic_tool_result(thread_id, turn_id, call_id, result).await {
    Ok(accepted) => Ok(accepted),
    Err(e) if e.to_string().contains("ended before acknowledgement") => {
        if shutting_down() { return Err(e); } // quiesce path: outcome unknown
        tracing::error!(%call_id, "settlement task dropped; inspect JSONL for terminal event");
        Err(e)
    }
    Err(e) => Err(e),
}

Prevention

Drain in-flight settlements (quiesce) before dropping the Runtime manager
Avoid tearing down the tokio runtime while submissions are outstanding
On missing ack, inspect the JSONL log for the terminal line before retrying - a landed append plus retry duplicates a receipt

When it happens

Trigger: The spawned settlement task (spawn_dynamic_tool_settlement) is dropped because the runtime is shutting down or the tokio runtime is torn down while a settlement is in flight; distinct from 635, which covers acknowledged failures. See runtime_threads.rs:4328.

Common situations: Process exit or runtime shutdown racing result submission; a test harness dropping the runtime mid-settlement; abrupt cancellation of the task's parent.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/81feb0bd9370f04d. Report an issue: GitHub.