coollabsio/coolify · warning · RuntimeException

Wait for the running storage backup and recovery operations

Error message

Wait for the running storage backup and recovery operations to finish before deleting this schedule.

What it means

Thrown by DeleteScheduledVolumeBackup after it acquires the per-schedule cache lock (VolumeBackupJob::lockKey) but finds at least one execution row with status='running', stop_recovery_pending=true, or s3_cleanup_pending=true. It prevents deleting a ScheduledVolumeBackup while a VolumeBackupJob is still mid-flight or has pending stop-recovery / S3-cleanup follow-ups. The schedule and its archives are only removed after all in-flight work drains.

Source

Thrown at app/Actions/Shared/DeleteScheduledVolumeBackup.php:30

{
    use AsAction;

    public function handle(ScheduledVolumeBackup $backup, ?Server $server = null): void
    {
        $lock = Cache::lock(VolumeBackupJob::lockKey($backup->id), $backup->timeout + 300);

        if (! $lock->get()) {
            throw new \RuntimeException('Wait for the queued or running storage backup to finish before deleting this schedule.');
        }

        try {
            if ($backup->executions()
                ->where(fn ($query) => $query
                    ->where('status', 'running')
                    ->orWhere('stop_recovery_pending', true)
                    ->orWhere('s3_cleanup_pending', true))
                ->exists()) {
                throw new \RuntimeException('Wait for the running storage backup and recovery operations to finish before deleting this schedule.');
            }

            $localFilenames = $backup->executions()
                ->where('local_storage_deleted', false)
                ->pluck('filename')
                ->filter()
                ->all();

            if ($localFilenames !== []) {
                $server ??= $backup->server();
                if (! $server) {
                    throw new \RuntimeException('The server is unavailable, so local backup archives cannot be deleted.');
                }

                deleteBackupsLocally($localFilenames, $server, throwError: true);
            }

            $s3Executions = $backup->executions()

View on GitHub (pinned to 70b9acc424)

Solutions

1. Wait for the running execution to finish (watch the executions list for the schedule) and retry the delete.
2. Check the queue (Horizon) for a queued or stuck VolumeBackupJob for this schedule id and let it complete or fail it cleanly.
3. If an execution is stale because its worker died (no live job but flags still set), mark that execution's status/stop_recovery_pending/s3_cleanup_pending as resolved, then delete again.
4. If you call this action programmatically, treat it as a transient conflict: surface 'try again later' instead of a hard failure.

Example fix

// before: delete fails while a backup runs
DeleteScheduledVolumeBackup::run($scheduledVolumeBackup);

// after: only delete when no in-flight work, otherwise retry later
$busy = $scheduledVolumeBackup->executions()
    ->where(fn ($q) => $q->where('status', 'running')
        ->orWhere('stop_recovery_pending', true)
        ->orWhere('s3_cleanup_pending', true))
    ->exists();

if ($busy) {
    // schedule a retry or inform the user; do not force-delete
    throw new \RuntimeException('Backup in progress, retry deletion later.');
}

DeleteScheduledVolumeBackup::run($scheduledVolumeBackup);

// Check for in-flight work before invoking the delete action
use App\Models\ScheduledVolumeBackup;

function scheduleHasPendingWork(ScheduledVolumeBackup $backup): bool
{
    return $backup->executions()
        ->where(fn ($q) => $q->where('status', 'running')
            ->orWhere('stop_recovery_pending', true)
            ->orWhere('s3_cleanup_pending', true))
        ->exists();
}

try {
    DeleteScheduledVolumeBackup::run($backup);
} catch (\RuntimeException $e) {
    if (str_contains($e->getMessage(), 'before deleting this schedule')) {
        // transient conflict: back off and retry after the execution drains
        Cache::lock(VolumeBackupJob::lockKey($backup->id), 5)->block(30);
        return retry(3, fn () => DeleteScheduledVolumeBackup::run($backup), 5000);
    }
    throw $e;
}

When it happens

Trigger: Calling the delete action for a volume backup schedule (UI delete button or API) while an execution has status='running'; deleting immediately after a stop request while stop_recovery_pending is still true; deleting while an S3 cleanup task flag (s3_cleanup_pending) is set on any execution.

Common situations: User clicks Delete during a large volume backup that takes minutes; a queued VolumeBackupJob picked up the schedule just before the delete request; a crashed queue worker left an execution permanently marked running/pending; slow S3 cleanup keeps the flag set longer than expected.

Related errors

• Invalid Cron / Human expression
• Wait for the queued or running storage backup to finish befo
• The server is unavailable, so local backup archives cannot b
• The S3 storage used by an existing backup is unavailable.
• SSH key file was not created: {$keyLocation}

AI-assisted analysis of coollabsio/coolify@70b9acc424 (2026-08-17). Data as JSON: /api/errors/9e2a3a83960ae726. Report an issue: GitHub.