{"record":{"id":"29ccc6d120085894","repo":"block/buzz","slug":"another-storage-snapshot-worker-already-holds-the-lease","errorCode":null,"errorMessage":"another storage-snapshot worker already holds the lease","messagePattern":"another storage-snapshot worker already holds the lease","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/buzz-admin/src/main.rs","lineNumber":191,"sourceCode":"        } => cmd_list_product_feedback(limit).await,\n        Command::Deletions { command } => deletions::run(command).await,\n        Command::ReconcileChannels { channel, relay_key } => {\n            reconcile_channels(channel, relay_key).await?;\n            Ok(0)\n        }\n    }\n}\n\nasync fn cmd_storage_snapshot(max_objects: u64) -> Result<i32> {\n    let max_objects_db = i64::try_from(max_objects)\n        .map_err(|_| anyhow::anyhow!(\"--max-objects must be at most {}\", i64::MAX))?;\n    if max_objects == 0 {\n        return Err(anyhow::anyhow!(\"--max-objects must be greater than zero\"));\n    }\n\n    let db = connect_db().await?;\n    let mut leader = db.try_lock_storage_accounting().await?.ok_or_else(|| {\n        anyhow::anyhow!(\"another storage-snapshot worker already holds the lease\")\n    })?;\n    let storage = Arc::new(MediaStorage::new(&storage_config_from_env()?)?);\n    let code_sha =\n        std::env::var(\"BUZZ_STORAGE_SNAPSHOT_CODE_SHA\").unwrap_or_else(|_| \"unknown\".to_string());\n    if code_sha.is_empty() || code_sha.len() > 128 {\n        return Err(anyhow::anyhow!(\n            \"BUZZ_STORAGE_SNAPSHOT_CODE_SHA must contain 1 to 128 bytes\"\n        ));\n    }\n\n    println!(\n        \"{}\",\n        serde_json::json!({\n            \"event\": \"storage_snapshot_started\",\n            \"max_objects\": max_objects,\n            \"code_sha\": code_sha,\n        })\n    );","sourceCodeStart":173,"sourceCodeEnd":209,"githubUrl":"https://github.com/block/buzz/blob/ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be/crates/buzz-admin/src/main.rs#L173-L209","documentation":"Storage snapshots are serialized via a database lease (try_lock_storage_accounting). If another worker currently holds the lease, this invocation returns immediately with this error instead of blocking or queueing, so only one snapshot writer runs at a time.","triggerScenarios":"Running `buzz-admin storage-snapshot` while a previous snapshot run (cron job, another operator, a stalled run that never released the lock) still holds the storage-accounting lease. try_lock_storage_accounting returns None and ok_or_else converts it to this error.","commonSituations":"Overlapping cron schedules, a second operator running the same command concurrently, or a crashed/killed run whose lease has not expired yet (stale lease).","solutions":["Wait for the existing snapshot to finish and retry, or remove the duplicate cron/schedule overlap.","Check for a stuck/stale lease from a crashed worker and clear it once confirmed no worker is alive.","Add distributed scheduling (e.g. flock or leader election) so only one instance of the command is launched.","Retry with backoff rather than tight-looping: the lease is transient if the other worker is healthy."],"exampleFix":"// before: cron every minute, overlapping runs\n* * * * * buzz-admin storage-snapshot --max-objects 1000000\n// after: widen the interval or gate with flock\n0 * * * * flock -n /tmp/buzz-snapshot.lock buzz-admin storage-snapshot --max-objects 1000000","handlingStrategy":"retry","validationCode":"// Best-effort pre-check: ensure no overlapping schedule is due now\n// (e.g. skip launch if a lockfile from a live run exists)\nif [ -e /tmp/buzz-snapshot.lock ] && kill -0 \"$(cat /tmp/buzz-snapshot.lock)\" 2>/dev/null; then\n  echo \"snapshot already running\"; exit 0\nfi","typeGuard":null,"tryCatchPattern":"match cmd_storage_snapshot(max_objects).await {\n    Err(e) if e.to_string().contains(\"already holds the lease\") => {\n        // schedule retry with backoff; do not tight-loop\n        tokio::time::sleep(Duration::from_secs(300)).await;\n    }\n    other => other?,\n}","preventionTips":["Serialize invocations with flock or a scheduler that prevents overlap.","Use exponential backoff with a terminal retry count, never tight retries.","After a crashed run, verify and clear stale leases before relaunching."],"tags":["concurrency","locking","storage","lease"],"backgroundTag":"conflicting-config-options","analyzedSha":"ef2aa1ae38fadcc0bc22b8bf6ed96b35933146be","analyzedAt":"2026-09-20T04:38:17.397Z","contentChangedAt":"2026-09-20T04:38:17.397Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}