{"record":{"id":"4cedd3a601dfe089","repo":"coollabsio/coolify","slug":"wait-for-the-queued-or-running-storage-backup-to-f","errorCode":null,"errorMessage":"Wait for the queued or running storage backup to finish before deleting this schedule.","messagePattern":"Wait for the queued or running storage backup to finish before deleting this schedule\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"warning","filePath":"app/Actions/Shared/DeleteScheduledVolumeBackup.php","lineNumber":20,"sourceCode":"\nnamespace App\\Actions\\Shared;\n\nuse App\\Jobs\\VolumeBackupJob;\nuse App\\Models\\ScheduledVolumeBackup;\nuse App\\Models\\Server;\nuse Illuminate\\Support\\Facades\\Cache;\nuse Lorisleiva\\Actions\\Concerns\\AsAction;\n\nclass DeleteScheduledVolumeBackup\n{\n    use AsAction;\n\n    public function handle(ScheduledVolumeBackup $backup, ?Server $server = null): void\n    {\n        $lock = Cache::lock(VolumeBackupJob::lockKey($backup->id), $backup->timeout + 300);\n\n        if (! $lock->get()) {\n            throw new \\RuntimeException('Wait for the queued or running storage backup to finish before deleting this schedule.');\n        }\n\n        try {\n            if ($backup->executions()\n                ->where(fn ($query) => $query\n                    ->where('status', 'running')\n                    ->orWhere('stop_recovery_pending', true)\n                    ->orWhere('s3_cleanup_pending', true))\n                ->exists()) {\n                throw new \\RuntimeException('Wait for the running storage backup and recovery operations to finish before deleting this schedule.');\n            }\n\n            $localFilenames = $backup->executions()\n                ->where('local_storage_deleted', false)\n                ->pluck('filename')\n                ->filter()\n                ->all();\n","sourceCodeStart":2,"sourceCodeEnd":38,"githubUrl":"https://github.com/coollabsio/coolify/blob/70b9acc42467278373e00de77abb40684e25b395/app/Actions/Shared/DeleteScheduledVolumeBackup.php#L2-L38","documentation":"DeleteScheduledVolumeBackup acquires the same atomic cache lock that VolumeBackupJob holds (VolumeBackupJob::lockKey($backup->id), job TTL timeout+60s, deleter TTL timeout+300s). If acquisition fails, a backup for this schedule is queued or running right now, and deleting the schedule concurrently would orphan running executions, so the action throws immediately. This is intentional serialized access, not corruption — the backup job will release the lock when it finishes.","triggerScenarios":"Deleting a scheduled volume backup while a VolumeBackupJob for the same schedule id is queued or executing (large backups hold the lock for up to the configured timeout, default 3600s); retrying deletion immediately after a previous failed attempt that still holds the lock briefly.","commonSituations":"Deleting a schedule during nightly backup windows; long S3 uploads; double-clicking delete causing overlapping requests.","solutions":["Wait for the running/queued backup to finish (check scheduled_volume_backup_executions status), then retry the delete.","If you must delete now, cancel/skip the running backup job first (or wait for its timeout to expire — the lock auto-expires after timeout+60).","If no job is visibly running, check for a stale lock from a crashed worker (redis CACHE_LOCK:* keys) — it expires on its own thanks to expireAfter."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// check for live work before attempting deletion\n$busy = $backup->executions()\n    ->where(fn ($q) => $q->where('status', 'running')\n        ->orWhere('stop_recovery_pending', true)\n        ->orWhere('s3_cleanup_pending', true))\n    ->exists();\nif ($busy) {\n    return 'Backup in progress — retry deletion after it completes.';\n}\n\\App\\Actions\\Shared\\DeleteScheduledVolumeBackup::run($backup);","typeGuard":null,"tryCatchPattern":"$attempt = fn () => \\App\\Actions\\Shared\\DeleteScheduledVolumeBackup::run($backup);\ntry {\n    $attempt();\n} catch (\\RuntimeException $e) {\n    if (str_contains($e->getMessage(), 'queued or running storage backup')) {\n        // transient lock contention: back off (e.g. 60s, exponential, capped) and retry;\n        // lock auto-expires after timeout+60s even if the worker died\n        retry(3, $attempt, 60000, throw: false) ?: report($e);\n    } else {\n        throw $e; // the other RuntimeExceptions here (missing server/S3) are permanent\n    }\n}","preventionTips":["Schedule deletions outside backup windows (avoid the nightly run slot).","Show the execution status (running/queued) in the UI and disable Delete while busy.","Rely on lock auto-expiry (timeout+60s) rather than manually purging cache locks after crashes.","Retry with backoff — never hammer the delete endpoint while a long S3 upload holds the lock."],"tags":["cache-lock","concurrency","backup-schedule","volume-backup","retry"],"backgroundTag":"concurrent-operation-lock","analyzedSha":"70b9acc42467278373e00de77abb40684e25b395","analyzedAt":"2026-08-17T01:41:01.313Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}