coollabsio/coolify · warning · RuntimeException

Wait for the queued or running storage backup to finish befo

Error message

Wait for the queued or running storage backup to finish before deleting this schedule.

What it means

DeleteScheduledVolumeBackup acquires the same atomic cache lock that VolumeBackupJob holds (VolumeBackupJob::lockKey($backup->id), job TTL timeout+60s, deleter TTL timeout+300s). If acquisition fails, a backup for this schedule is queued or running right now, and deleting the schedule concurrently would orphan running executions, so the action throws immediately. This is intentional serialized access, not corruption — the backup job will release the lock when it finishes.

Source

Thrown at app/Actions/Shared/DeleteScheduledVolumeBackup.php:20

namespace App\Actions\Shared;

use App\Jobs\VolumeBackupJob;
use App\Models\ScheduledVolumeBackup;
use App\Models\Server;
use Illuminate\Support\Facades\Cache;
use Lorisleiva\Actions\Concerns\AsAction;

class DeleteScheduledVolumeBackup
{
    use AsAction;

    public function handle(ScheduledVolumeBackup $backup, ?Server $server = null): void
    {
        $lock = Cache::lock(VolumeBackupJob::lockKey($backup->id), $backup->timeout + 300);

        if (! $lock->get()) {
            throw new \RuntimeException('Wait for the queued or running storage backup to finish before deleting this schedule.');
        }

        try {
            if ($backup->executions()
                ->where(fn ($query) => $query
                    ->where('status', 'running')
                    ->orWhere('stop_recovery_pending', true)
                    ->orWhere('s3_cleanup_pending', true))
                ->exists()) {
                throw new \RuntimeException('Wait for the running storage backup and recovery operations to finish before deleting this schedule.');
            }

            $localFilenames = $backup->executions()
                ->where('local_storage_deleted', false)
                ->pluck('filename')
                ->filter()
                ->all();

View on GitHub (pinned to 70b9acc424)

Solutions

Wait for the running/queued backup to finish (check scheduled_volume_backup_executions status), then retry the delete.
If you must delete now, cancel/skip the running backup job first (or wait for its timeout to expire — the lock auto-expires after timeout+60).
If no job is visibly running, check for a stale lock from a crashed worker (redis CACHE_LOCK:* keys) — it expires on its own thanks to expireAfter.

Defensive patterns

Strategy: retry

Validate before calling

// check for live work before attempting deletion
$busy = $backup->executions()
    ->where(fn ($q) => $q->where('status', 'running')
        ->orWhere('stop_recovery_pending', true)
        ->orWhere('s3_cleanup_pending', true))
    ->exists();
if ($busy) {
    return 'Backup in progress — retry deletion after it completes.';
}
\App\Actions\Shared\DeleteScheduledVolumeBackup::run($backup);

Try / catch

$attempt = fn () => \App\Actions\Shared\DeleteScheduledVolumeBackup::run($backup);
try {
    $attempt();
} catch (\RuntimeException $e) {
    if (str_contains($e->getMessage(), 'queued or running storage backup')) {
        // transient lock contention: back off (e.g. 60s, exponential, capped) and retry;
        // lock auto-expires after timeout+60s even if the worker died
        retry(3, $attempt, 60000, throw: false) ?: report($e);
    } else {
        throw $e; // the other RuntimeExceptions here (missing server/S3) are permanent
    }
}

Prevention

Schedule deletions outside backup windows (avoid the nightly run slot).
Show the execution status (running/queued) in the UI and disable Delete while busy.
Rely on lock auto-expiry (timeout+60s) rather than manually purging cache locks after crashes.
Retry with backoff — never hammer the delete endpoint while a long S3 upload holds the lock.

When it happens

Trigger: Deleting a scheduled volume backup while a VolumeBackupJob for the same schedule id is queued or executing (large backups hold the lock for up to the configured timeout, default 3600s); retrying deletion immediately after a previous failed attempt that still holds the lock briefly.

Common situations: Deleting a schedule during nightly backup windows; long S3 uploads; double-clicking delete causing overlapping requests.

Related errors

AI-assisted analysis of coollabsio/coolify@70b9acc424 (2026-08-17). Data as JSON: /api/errors/4cedd3a601dfe089. Report an issue: GitHub.