{"record":{"id":"6d2b58ba10dc122d","repo":"zeroclaw-labs/zeroclaw","slug":"mcp-server-server-name-recovery-task-failed-be","errorCode":null,"errorMessage":"MCP server `{server_name}` recovery task failed before writing {operation}","messagePattern":"MCP server `(.+?)` recovery task failed before writing (.+?)","errorType":"exception","errorClass":"JoinError","httpStatus":null,"severity":"error","filePath":"crates/zeroclaw-tools/src/mcp_client.rs","lineNumber":696,"sourceCode":"                        self.spawn_recovery(epoch, operation.to_string());\n                        return Err(error).with_context(|| {\n                            format!(\n                                \"MCP server `{server_name}` failed during {operation}; outcome \\\n                                 unknown and request was not replayed\"\n                            )\n                        });\n                    }\n\n                    cancellation_guard.disarm();\n                    let recoverable = error.downcast_ref::<McpTransportError>().is_some();\n                    if recoverable && pre_write_retries < MAX_RECONNECT_ATTEMPTS {\n                        pre_write_retries += 1;\n                        let observed_epoch = lifecycle.pre_write_epoch().unwrap_or(0);\n                        let recovery = self.start_recovery(observed_epoch, operation.to_string());\n                        match timeout_at(deadline, recovery).await {\n                            Ok(Ok(result)) => result?,\n                            Ok(Err(join_error)) => {\n                                return Err(anyhow::Error::new(join_error)).with_context(|| {\n                                    format!(\n                                        \"MCP server `{server_name}` recovery task failed before \\\n                                         writing {operation}\"\n                                    )\n                                });\n                            }\n                            Err(_) => {\n                                bail!(\n                                    \"MCP server `{server_name}` exhausted the {timeout_secs}s \\\n                                     budget recovering before writing {operation}\"\n                                );\n                            }\n                        }\n                        continue;\n                    }\n                    return Err(error).with_context(|| {\n                        format!(\"MCP server `{server_name}` error during {operation}\")\n                    });","sourceCodeStart":678,"sourceCodeEnd":714,"githubUrl":"https://github.com/zeroclaw-labs/zeroclaw/blob/88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc/crates/zeroclaw-tools/src/mcp_client.rs#L678-L714","documentation":"When dispatch_rpc hits a recoverable McpTransportError before the JSON-RPC request was written, it calls start_recovery (transport reset + re-handshake, epoch-guarded) and awaits that JoinHandle under the remaining deadline; if the handle resolves with a tokio JoinError — the recovery task panicked or was cancelled — the JoinError is wrapped with this context (mcp_client.rs:695-702). \"before writing {operation}\" is the safety claim: the request never reached the server, so retrying the whole call is side-effect-free. Cancellation most often happens at runtime shutdown; panics come from bugs or poisoned state inside the recovery path itself.","triggerScenarios":"call_tool or dispatch_method on an MCP server whose transport just dropped (server process restarted), where the spawned recovery task then panics or the tokio runtime shuts down mid-recovery (JoinError::is_cancelled), within the MAX_RECONNECT_ATTEMPTS pre-write retry loop.","commonSituations":"MCP stdio server process crashing while the daemon is shutting down (recovery races runtime teardown); a panic inside recovery from a poisoned lock; aggressive test harnesses dropping the runtime while a call is in flight.","solutions":["Retry the tool call — the failure happened before the request was written, so no server-side side effect occurred","Check logs for the preceding panic line from the recovery task and fix or report the panic source","If it occurs at shutdown, fix shutdown ordering: quiesce MCP client activity before dropping the tokio runtime","Verify the MCP server process itself is healthy/restartable (command path, env, working dir) so recovery can succeed on the next attempt"],"exampleFix":"// before\nlet value = client.call_tool(\"search\", args).await?; // one shot; join failure is fatal to the flow\n\n// after — pre-write failures are safe to retry\nfor _ in 0..3 {\n    match client.call_tool(\"search\", args.clone()).await {\n        Ok(v) => return Ok(v),\n        Err(e) if e.chain().any(|c| c.downcast_ref::<tokio::task::JoinError>().is_some()) => {\n            continue; // recovery task died before the request was written\n        }\n        Err(e) => return Err(e),\n    }\n}\nanyhow::bail!(\"mcp call kept failing in pre-write recovery\");","handlingStrategy":"retry","validationCode":null,"typeGuard":"fn is_recovery_join_failure(e: &anyhow::Error) -> bool {\n    e.chain().any(|c| c.downcast_ref::<tokio::task::JoinError>().is_some())\n}","tryCatchPattern":"match client.call_tool(tool, args).await {\n    Ok(v) => v,\n    Err(e) if is_recovery_join_failure(&e) => retry_with_backoff().await, // pre-write: safe\n    Err(e) => return Err(e),\n}","preventionTips":["Order shutdown: stop issuing MCP calls and await in-flight ones before dropping the tokio runtime","Distinguish pre-write failures (safe to retry) from outcome-unknown ones (the library already refuses to replay those)","Watch MCP server process health; recovery can only succeed if the server can restart","Log JoinError::is_panic vs is_cancelled to separate bugs from shutdown races"],"tags":["rust","mcp","json-rpc","tokio","join-error","recovery","retry"],"backgroundTag":"background-task-join-failure","analyzedSha":"88bb9c8533fc57ed7a03e36ca7c9ed2bf8336dcc","analyzedAt":"2026-08-23T01:07:41.857Z","schemaVersion":2},"datasetVersion":"2026-08-23T08:06:27.607Z"}