block/buzz · error · anyhow::Error

Shutdown task failed

Error message

Shutdown task failed: {e}

What it means

After the TCP server future completes (normal shutdown), serve() awaits the JoinHandle of the spawned shutdown task — the one that runs the jittered drain of live WebSocket connections (ConnectionManager::drain_all_jittered within GRACEFUL_DRAIN_TIMEOUT, 30s) and arms the hard_shutdown_abort. A JoinError here means that task panicked or was cancelled before completing, so the drain bookkeeping did not finish; the message distinguishes it from the server error.

Solutions

1. Reproduce the shutdown path: send SIGTERM (`kill -TERM <pid>`), capture the panic backtrace with `RUST_BACKTRACE=1` — the panic location inside the drain task is the real defect.
2. Inspect JoinError::is_panic() vs is_cancelled(): panic = code bug in drain; cancelled = runtime dropped during drain (grace period too short under supervisor like `docker stop -t 5`).
3. Increase the supervisor's stop grace period above GRACEFUL_DRAIN_TIMEOUT (30s) so the drain task can finish instead of being cancelled.
4. If the panic is in drain_all_jittered, fix that call site (no unwrap/expect on the connection registry path) — main.rs already has a TODO(coverage) noting this wiring lacks automated tests; add one.

// separate panic from cancellation so the response differs
let hard_shutdown = match shutdown_handle.await {
    Ok(h) => h,
    Err(e) if e.is_panic() => {
        tracing::error!(panic = ?e.into_panic(), "drain task panicked");
        return Err(anyhow::anyhow!("Shutdown task failed: {e}"));
    }
    Err(e) => {
        tracing::warn!("drain task cancelled: {e}");
        return Ok(());
    }
};

When it happens

Trigger: shutdown_handle.await at crates/buzz-relay/src/main.rs:1375-1377 returns JoinError when the drain task panics — e.g. a bug or unwrap/expect inside drain_all_jittered, a poisoned lock on the connection registry, or cancellation via runtime shutdown while the drain was in flight (JoinError::is_cancelled).

Common situations: A regression in ConnectionManager drain logic that unwinds; a blocking mutex deadlock that turns into cancellation on runtime drop; SIGKILL-adjacent shutdowns where the runtime is torn down before the 30s drain finishes; test harnesses aborting the tokio runtime at grace-period expiry.

Related errors

• install SIGTERM handler
• pool initialization cancelled by shutdown
• ban failed
• ban requires target_pubkey
• bundled model-capabilities.json failed validation

AI-assisted analysis of block/buzz@ef2aa1ae38 (2026-08-20). Data as JSON: /api/errors/fb5f6ee6a6d906a5. Report an issue: GitHub.

Appendix: source

Thrown at crates/buzz-relay/src/main.rs:1493

                })
                .await
                .ok();
        });

        let mut tcp_rx = shutdown_tx.subscribe();
        axum::serve(
            tcp_listener,
            router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
        )
        .with_graceful_shutdown(async move {
            tcp_rx.changed().await.ok();
        })
        .await
        .map_err(|e| anyhow::anyhow!("TCP server error: {e}"))?;

        let hard_shutdown = shutdown_handle
            .await
            .map_err(|e| anyhow::anyhow!("Shutdown task failed: {e}"))?;
        uds_handle.abort();
        hard_shutdown.abort();
        return Ok(());
    }

    #[cfg(not(unix))]
    if config.uds_path.is_some() {
        tracing::warn!("BUZZ_UDS_PATH set but UDS not supported on this platform");
    }

    // TCP-only path.
    let mut tcp_rx = shutdown_tx.subscribe();
    axum::serve(
        tcp_listener,
        router.into_make_service_with_connect_info::<std::net::SocketAddr>(),
    )
    .with_graceful_shutdown(async move {
        tcp_rx.changed().await.ok();

View on GitHub (pinned to ef2aa1ae38)