{"record":{"id":"7827f5192cfff30a","repo":"block/buzz","slug":"deletion-heartbeat-task-failed-error","errorCode":null,"errorMessage":"deletion heartbeat task failed: {error}","messagePattern":"deletion heartbeat task failed: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/buzz-deletion/src/lib.rs","lineNumber":1006,"sourceCode":"                        heartbeat_error_signal.cancel();\n                        return;\n                    }\n                }\n            }\n        }\n    });\n\n    let stage = await_stage(\n        execute_stage(services, claim, &heartbeat_error),\n        shutdown,\n        &heartbeat_error,\n    )\n    .await;\n    heartbeat_shutdown.cancel();\n    match heartbeat.await {\n        Ok(()) => stage,\n        Err(error) => {\n            StageOutcome::Failed(anyhow::anyhow!(\"deletion heartbeat task failed: {error}\"))\n        }\n    }\n}\n\nasync fn run_guarded_external_step<F, Fut, T>(\n    services: &Services,\n    token: &LeaseToken,\n    stage: DeletionStage,\n    heartbeat_lost: &CancellationToken,\n    operation: F,\n) -> Result<T>\nwhere\n    F: FnOnce() -> Fut,\n    Fut: std::future::Future<Output = Result<T>>,\n{\n    services.store.verify_execution_token(token, stage).await?;\n    let result = tokio::select! {\n        biased;","sourceCodeStart":988,"sourceCodeEnd":1024,"githubUrl":"https://github.com/block/buzz/blob/f956e6fe06a76e50cbd8fba1a162482e752e7f1a/crates/buzz-deletion/src/lib.rs#L988-L1024","documentation":"run_stage_with_heartbeat() spawns a background task that periodically renews the deletion lease (store.heartbeat) while the main stage executes; a heartbeat failure normally signals loss via a CancellationToken (surfacing elsewhere as DeletionLeaseLost). THIS error is different: heartbeat.await itself returned Err, i.e. a tokio JoinError — the heartbeat task was aborted/panicked rather than exiting cleanly. Its Ok path (including heartbeat DB errors) returns normally, so this message means the renewal loop crashed unexpectedly ({error} is the JoinError detail, e.g. task panicked), and the stage outcome is replaced with this failure.","triggerScenarios":"A panic inside the heartbeat loop (e.g. a store/driver invariant violated during heartbeat), or task abortion at runtime shutdown while a stage is in flight. Redis/DB connectivity problems during heartbeat do NOT produce this — they take the cancellation-token path (DeletionLeaseLost) instead.","commonSituations":"Rare in practice: driver-level panics (deadpool/redis client bug), a runtime shutting down mid-deletion, or a custom build with a patched tokio/store. If reproducible, it indicates a bug in the heartbeat path rather than an operational misconfiguration.","solutions":["Re-run the executor: the lease fencing means a crashed heartbeat loses the lease, and after DEFAULT_LEASE_DURATION expires the request can be re-claimed and execution resumes from the recorded stage.","Capture the {error} suffix — 'task <id> panicked' points at a code bug; look for the panic message in stderr/logs just above this error.","If it recurs at the same stage, check store/driver versions (tokio, deadpool-redis, sqlx) against the workspace lockfile and report the panic backtrace with the deletion request id.","Verify no single stage outlives many lease durations — long stages rely entirely on this heartbeat, so a flaky one that panics repeatedly will wedge progress."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"# JoinError is not deterministic: retry via re-claim after lease expiry\n# fetch run output; nonzero exit with 'heartbeat task failed' => re-run executor later\nuntil buzz-deletion run --id \"$RID\"; do echo \"heartbeat failed; lease will expire, retrying\" >&2; sleep 90; done","preventionTips":["Keep stages reasonably short and idempotent so lease-expiry retries always make progress.","Pin tokio/deadpool/sqlx versions via Cargo.lock; heartbeat panics usually indicate dependency bugs.","Capture stderr around the failure — the panic message preceding this error is the real diagnostic."],"tags":["buzz-deletion","heartbeat","tokio","lease-renewal","task-panic"],"backgroundTag":"background-task-panic","analyzedSha":"f956e6fe06a76e50cbd8fba1a162482e752e7f1a","analyzedAt":"2026-08-16T22:11:40.750Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}