zeroclaw-labs/zeroclaw · error · JoinError

MCP server `{server_name}` recovery task failed before writi

Error message

MCP server `{server_name}` recovery task failed before writing {operation}

What it means

When dispatch_rpc hits a recoverable McpTransportError before the JSON-RPC request was written, it calls start_recovery (transport reset + re-handshake, epoch-guarded) and awaits that JoinHandle under the remaining deadline; if the handle resolves with a tokio JoinError — the recovery task panicked or was cancelled — the JoinError is wrapped with this context (mcp_client.rs:695-702). "before writing {operation}" is the safety claim: the request never reached the server, so retrying the whole call is side-effect-free. Cancellation most often happens at runtime shutdown; panics come from bugs or poisoned state inside the recovery path itself.

Source

Thrown at crates/zeroclaw-tools/src/mcp_client.rs:696

                        self.spawn_recovery(epoch, operation.to_string());
                        return Err(error).with_context(|| {
                            format!(
                                "MCP server `{server_name}` failed during {operation}; outcome \
                                 unknown and request was not replayed"
                            )
                        });
                    }

                    cancellation_guard.disarm();
                    let recoverable = error.downcast_ref::<McpTransportError>().is_some();
                    if recoverable && pre_write_retries < MAX_RECONNECT_ATTEMPTS {
                        pre_write_retries += 1;
                        let observed_epoch = lifecycle.pre_write_epoch().unwrap_or(0);
                        let recovery = self.start_recovery(observed_epoch, operation.to_string());
                        match timeout_at(deadline, recovery).await {
                            Ok(Ok(result)) => result?,
                            Ok(Err(join_error)) => {
                                return Err(anyhow::Error::new(join_error)).with_context(|| {
                                    format!(
                                        "MCP server `{server_name}` recovery task failed before \
                                         writing {operation}"
                                    )
                                });
                            }
                            Err(_) => {
                                bail!(
                                    "MCP server `{server_name}` exhausted the {timeout_secs}s \
                                     budget recovering before writing {operation}"
                                );
                            }
                        }
                        continue;
                    }
                    return Err(error).with_context(|| {
                        format!("MCP server `{server_name}` error during {operation}")
                    });

View on GitHub (pinned to 88bb9c8533)

Solutions

Retry the tool call — the failure happened before the request was written, so no server-side side effect occurred
Check logs for the preceding panic line from the recovery task and fix or report the panic source
If it occurs at shutdown, fix shutdown ordering: quiesce MCP client activity before dropping the tokio runtime
Verify the MCP server process itself is healthy/restartable (command path, env, working dir) so recovery can succeed on the next attempt

Example fix

// before
let value = client.call_tool("search", args).await?; // one shot; join failure is fatal to the flow

// after — pre-write failures are safe to retry
for _ in 0..3 {
    match client.call_tool("search", args.clone()).await {
        Ok(v) => return Ok(v),
        Err(e) if e.chain().any(|c| c.downcast_ref::<tokio::task::JoinError>().is_some()) => {
            continue; // recovery task died before the request was written
        }
        Err(e) => return Err(e),
    }
}
anyhow::bail!("mcp call kept failing in pre-write recovery");

Defensive patterns

Strategy: retry

Type guard

fn is_recovery_join_failure(e: &anyhow::Error) -> bool {
    e.chain().any(|c| c.downcast_ref::<tokio::task::JoinError>().is_some())
}

Try / catch

match client.call_tool(tool, args).await {
    Ok(v) => v,
    Err(e) if is_recovery_join_failure(&e) => retry_with_backoff().await, // pre-write: safe
    Err(e) => return Err(e),
}

Prevention

Order shutdown: stop issuing MCP calls and await in-flight ones before dropping the tokio runtime
Distinguish pre-write failures (safe to retry) from outcome-unknown ones (the library already refuses to replay those)
Watch MCP server process health; recovery can only succeed if the server can restart
Log JoinError::is_panic vs is_cancelled to separate bugs from shutdown races

When it happens

Trigger: call_tool or dispatch_method on an MCP server whose transport just dropped (server process restarted), where the spawned recovery task then panics or the tokio runtime shuts down mid-recovery (JoinError::is_cancelled), within the MAX_RECONNECT_ATTEMPTS pre-write retry loop.

Common situations: MCP stdio server process crashing while the daemon is shutting down (recovery races runtime teardown); a panic inside recovery from a poisoned lock; aggressive test harnesses dropping the runtime while a call is in flight.

Related errors

AI-assisted analysis of zeroclaw-labs/zeroclaw@88bb9c8533 (2026-08-23). Data as JSON: /api/errors/6d2b58ba10dc122d. Report an issue: GitHub.