{"record":{"id":"fb5f6ee6a6d906a5","repo":"block/buzz","slug":"shutdown-task-failed-e","errorCode":null,"errorMessage":"Shutdown task failed: {e}","messagePattern":"Shutdown task failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/buzz-relay/src/main.rs","lineNumber":1377,"sourceCode":"                })\n                .await\n                .ok();\n        });\n\n        let mut tcp_rx = shutdown_tx.subscribe();\n        axum::serve(\n            tcp_listener,\n            router.into_make_service_with_connect_info::<std::net::SocketAddr>(),\n        )\n        .with_graceful_shutdown(async move {\n            tcp_rx.changed().await.ok();\n        })\n        .await\n        .map_err(|e| anyhow::anyhow!(\"TCP server error: {e}\"))?;\n\n        let hard_shutdown = shutdown_handle\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Shutdown task failed: {e}\"))?;\n        uds_handle.abort();\n        hard_shutdown.abort();\n        return Ok(());\n    }\n\n    #[cfg(not(unix))]\n    if config.uds_path.is_some() {\n        tracing::warn!(\"BUZZ_UDS_PATH set but UDS not supported on this platform\");\n    }\n\n    // TCP-only path.\n    let mut tcp_rx = shutdown_tx.subscribe();\n    axum::serve(\n        tcp_listener,\n        router.into_make_service_with_connect_info::<std::net::SocketAddr>(),\n    )\n    .with_graceful_shutdown(async move {\n        tcp_rx.changed().await.ok();","sourceCodeStart":1359,"sourceCodeEnd":1395,"githubUrl":"https://github.com/block/buzz/blob/f956e6fe06a76e50cbd8fba1a162482e752e7f1a/crates/buzz-relay/src/main.rs#L1359-L1395","documentation":"serve() spawns a graceful-shutdown task (main.rs:1295): wait for SIGTERM/Ctrl+C, flip shutting_down, sleep a 5s grace, broadcast stop to all listeners, arm a 30s hard-exit timer (GRACEFUL_DRAIN_TIMEOUT), then drain every live WebSocket connection (jittered or all-at-once) and return the hard timer's abort handle. This error means shutdown_handle.await resolved to Err(JoinError): the task panicked — e.g. the .expect(\"install SIGTERM handler\") on SignalKind::terminate(), or a bug inside the drain helpers — or it was cancelled, so the drain result and abort handle were lost. It is a bug/runtime indicator, not a configuration error. Only reachable on the UDS+TCP path (BUZZ_UDS_PATH set; uds_handle.abort() follows this line).","triggerScenarios":"A panic inside the shutdown task: installing the SIGTERM handler fails (runtime or sandbox without unix signal support, double handler registration); a panic in ConnectionManager::drain_all_jittered during the 1012-close sequence; an embedder aborting the task. Typically observed right after sending SIGTERM to a UDS-configured relay.","commonSituations":"Almost always a relay defect or an exotic embedding (custom tokio runtime without signal support). The process exits with this error instead of completing a clean drain; connections may not have received their 1012 close frames.","solutions":["Scan upward in the logs for the panic message — JoinError carries the payload and it names the failing line (signal install vs drain).","Check how the binary is run: embedders must use a standard tokio runtime with unix signal support rather than a stripped-down runtime.","Update buzz-relay and report the panic payload — a panic in the shutdown path is a defect, not an operator error.","Operationally, rely on the supervisor's kill timeout (and the 30s hard-exit backstop if the task died after arming it) rather than expecting a clean drain."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_task_panic(e: &tokio::task::JoinError) -> bool {\n    e.is_panic()\n}\n\nfn is_task_cancelled(e: &tokio::task::JoinError) -> bool {\n    e.is_cancelled()\n}","tryCatchPattern":"match shutdown_handle.await {\n    Ok(abort_handle) => {\n        abort_handle.abort(); // drain finished; cancel the 30s hard-exit timer\n    }\n    Err(e) if e.is_panic() => {\n        let payload = e.into_panic();\n        tracing::error!(?payload, \"shutdown task panicked — connections may not have drained\");\n        std::process::exit(1);\n    }\n    Err(e) => return Err(anyhow!(\"Shutdown task failed: {e}\")),\n}","preventionTips":["Run the relay via its own main with a standard tokio runtime (unix signal support)","Do not abort the shutdown task yourself — serve() owns its lifecycle","Keep panic=abort off for this binary so the JoinError is observable","Exercise SIGTERM drains in staging before shipping shutdown-path changes"],"tags":["rust","tokio","joinerror","shutdown","panic","graceful-shutdown"],"backgroundTag":"tokio-task-panic","analyzedSha":"f956e6fe06a76e50cbd8fba1a162482e752e7f1a","analyzedAt":"2026-08-16T22:11:40.750Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}