block/buzz · error · anyhow::Error

deletion heartbeat task failed: {error}

Error message

deletion heartbeat task failed: {error}

What it means

run_stage_with_heartbeat() spawns a background task that periodically renews the deletion lease (store.heartbeat) while the main stage executes; a heartbeat failure normally signals loss via a CancellationToken (surfacing elsewhere as DeletionLeaseLost). THIS error is different: heartbeat.await itself returned Err, i.e. a tokio JoinError — the heartbeat task was aborted/panicked rather than exiting cleanly. Its Ok path (including heartbeat DB errors) returns normally, so this message means the renewal loop crashed unexpectedly ({error} is the JoinError detail, e.g. task panicked), and the stage outcome is replaced with this failure.

Source

Thrown at crates/buzz-deletion/src/lib.rs:1006

                        heartbeat_error_signal.cancel();
                        return;
                    }
                }
            }
        }
    });

    let stage = await_stage(
        execute_stage(services, claim, &heartbeat_error),
        shutdown,
        &heartbeat_error,
    )
    .await;
    heartbeat_shutdown.cancel();
    match heartbeat.await {
        Ok(()) => stage,
        Err(error) => {
            StageOutcome::Failed(anyhow::anyhow!("deletion heartbeat task failed: {error}"))
        }
    }
}

async fn run_guarded_external_step<F, Fut, T>(
    services: &Services,
    token: &LeaseToken,
    stage: DeletionStage,
    heartbeat_lost: &CancellationToken,
    operation: F,
) -> Result<T>
where
    F: FnOnce() -> Fut,
    Fut: std::future::Future<Output = Result<T>>,
{
    services.store.verify_execution_token(token, stage).await?;
    let result = tokio::select! {
        biased;

View on GitHub (pinned to f956e6fe06)

Solutions

1. Re-run the executor: the lease fencing means a crashed heartbeat loses the lease, and after DEFAULT_LEASE_DURATION expires the request can be re-claimed and execution resumes from the recorded stage.
2. Capture the {error} suffix — 'task <id> panicked' points at a code bug; look for the panic message in stderr/logs just above this error.
3. If it recurs at the same stage, check store/driver versions (tokio, deadpool-redis, sqlx) against the workspace lockfile and report the panic backtrace with the deletion request id.
4. Verify no single stage outlives many lease durations — long stages rely entirely on this heartbeat, so a flaky one that panics repeatedly will wedge progress.

# JoinError is not deterministic: retry via re-claim after lease expiry
# fetch run output; nonzero exit with 'heartbeat task failed' => re-run executor later
until buzz-deletion run --id "$RID"; do echo "heartbeat failed; lease will expire, retrying" >&2; sleep 90; done

When it happens

Trigger: A panic inside the heartbeat loop (e.g. a store/driver invariant violated during heartbeat), or task abortion at runtime shutdown while a stage is in flight. Redis/DB connectivity problems during heartbeat do NOT produce this — they take the cancellation-token path (DeletionLeaseLost) instead.

Common situations: Rare in practice: driver-level panics (deadpool/redis client bug), a runtime shutting down mid-deletion, or a custom build with a patched tokio/store. If reproducible, it indicates a bug in the heartbeat path rather than an operational misconfiguration.

Related errors

• --host must not be empty
• cannot derive community host; pass --host or set RELAY_URL
• cannot derive community host from RELAY_URL; pass --host or
• {name} is required for community deletion
• deletion request is not runnable, is blocked, or is leased b

AI-assisted analysis of block/buzz@f956e6fe06 (2026-08-16). Data as JSON: /api/errors/7827f5192cfff30a. Report an issue: GitHub.