{"record":{"id":"fb5f6ee6a6d906a5","repo":"block/buzz","slug":"shutdown-task-failed-e","errorCode":null,"errorMessage":"Shutdown task failed: {e}","messagePattern":"Shutdown task failed: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/buzz-relay/src/main.rs","lineNumber":1493,"sourceCode":"                })\n                .await\n                .ok();\n        });\n\n        let mut tcp_rx = shutdown_tx.subscribe();\n        axum::serve(\n            tcp_listener,\n            router.into_make_service_with_connect_info::<std::net::SocketAddr>(),\n        )\n        .with_graceful_shutdown(async move {\n            tcp_rx.changed().await.ok();\n        })\n        .await\n        .map_err(|e| anyhow::anyhow!(\"TCP server error: {e}\"))?;\n\n        let hard_shutdown = shutdown_handle\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Shutdown task failed: {e}\"))?;\n        uds_handle.abort();\n        hard_shutdown.abort();\n        return Ok(());\n    }\n\n    #[cfg(not(unix))]\n    if config.uds_path.is_some() {\n        tracing::warn!(\"BUZZ_UDS_PATH set but UDS not supported on this platform\");\n    }\n\n    // TCP-only path.\n    let mut tcp_rx = shutdown_tx.subscribe();\n    axum::serve(\n        tcp_listener,\n        router.into_make_service_with_connect_info::<std::net::SocketAddr>(),\n    )\n    .with_graceful_shutdown(async move {\n        tcp_rx.changed().await.ok();","sourceCodeStart":1475,"sourceCodeEnd":1511,"githubUrl":"https://github.com/block/buzz/blob/ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be/crates/buzz-relay/src/main.rs#L1475-L1511","documentation":"After the TCP server future completes (normal shutdown), serve() awaits the JoinHandle of the spawned shutdown task — the one that runs the jittered drain of live WebSocket connections (ConnectionManager::drain_all_jittered within GRACEFUL_DRAIN_TIMEOUT, 30s) and arms the hard_shutdown_abort. A JoinError here means that task panicked or was cancelled before completing, so the drain bookkeeping did not finish; the message distinguishes it from the server error.","triggerScenarios":"shutdown_handle.await at crates/buzz-relay/src/main.rs:1375-1377 returns JoinError when the drain task panics — e.g. a bug or unwrap/expect inside drain_all_jittered, a poisoned lock on the connection registry, or cancellation via runtime shutdown while the drain was in flight (JoinError::is_cancelled).","commonSituations":"A regression in ConnectionManager drain logic that unwinds; a blocking mutex deadlock that turns into cancellation on runtime drop; SIGKILL-adjacent shutdowns where the runtime is torn down before the 30s drain finishes; test harnesses aborting the tokio runtime at grace-period expiry.","solutions":["Reproduce the shutdown path: send SIGTERM (`kill -TERM <pid>`), capture the panic backtrace with `RUST_BACKTRACE=1` — the panic location inside the drain task is the real defect.","Inspect JoinError::is_panic() vs is_cancelled(): panic = code bug in drain; cancelled = runtime dropped during drain (grace period too short under supervisor like `docker stop -t 5`).","Increase the supervisor's stop grace period above GRACEFUL_DRAIN_TIMEOUT (30s) so the drain task can finish instead of being cancelled.","If the panic is in drain_all_jittered, fix that call site (no unwrap/expect on the connection registry path) — main.rs already has a TODO(coverage) noting this wiring lacks automated tests; add one."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// separate panic from cancellation so the response differs\nlet hard_shutdown = match shutdown_handle.await {\n    Ok(h) => h,\n    Err(e) if e.is_panic() => {\n        tracing::error!(panic = ?e.into_panic(), \"drain task panicked\");\n        return Err(anyhow::anyhow!(\"Shutdown task failed: {e}\"));\n    }\n    Err(e) => {\n        tracing::warn!(\"drain task cancelled: {e}\");\n        return Ok(());\n    }\n};","preventionTips":["Keep the drain task unwrap/expect-free (repo rule: no new unwrap in production paths).","Size supervisor stop grace periods above GRACEFUL_DRAIN_TIMEOUT (30s) plus drain jitter.","Exercise SIGTERM shutdown in CI/staging regularly — the code itself notes a coverage TODO for this wiring.","Treat any is_panic() JoinError as a code defect to fix, not an operational alarm to silence."],"tags":["rust","async","tokio","shutdown","drain","task-join"],"backgroundTag":"async-task-panic","analyzedSha":"ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be","analyzedAt":"2026-08-20T04:38:24.874Z","contentChangedAt":"2026-08-20T04:38:24.874Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}