coollabsio/coolify · critical · RuntimeException

The server is unavailable for container recovery.

Error message

The server is unavailable for container recovery.

What it means

After a volume backup that stopped containers (stop_recovery_pending), VolumeBackupRecoveryJob restarts them from the saved container ids. recoverContainers() resolves the server through `$execution->scheduledVolumeBackup?->server()`; if the scheduled backup or its server can no longer be resolved, it throws - the containers stopped for the backup remain stopped.

Source

Thrown at app/Jobs/VolumeBackupRecoveryJob.php:65

    public static function recover(ScheduledVolumeBackupExecution $execution): void
    {
        $execution->loadMissing('scheduledVolumeBackup.backupable.resource');

        if ($execution->stop_recovery_pending) {
            self::recoverContainers($execution);
        }

        if ($execution->s3_cleanup_pending) {
            self::cleanupS3Upload($execution);
        }
    }

    private static function recoverContainers(ScheduledVolumeBackupExecution $execution): void
    {
        $server = $execution->scheduledVolumeBackup?->server();

        if (! $server) {
            throw new \RuntimeException('The server is unavailable for container recovery.');
        }

        $stateFile = self::stateFile($execution);
        $output = instant_remote_process(
            ['cat '.escapeshellarg($stateFile).' 2>/dev/null || true'],
            $server,
            disableMultiplexing: true,
        );
        $containers = collect(preg_split('/\s+/', trim((string) $output)))
            ->filter(fn (string $container): bool => preg_match('/^[a-f0-9]{6,64}$/i', $container) === 1)
            ->values()
            ->all();
        $execution->update(['stop_container_ids' => $containers]);

        $remainingFile = $stateFile.'.remaining';
        $script = 'status=0; : > '.escapeshellarg($remainingFile).'; '
            .'if [ -f '.escapeshellarg($stateFile).' ]; then while IFS= read -r container; do '
            .'[ -z "$container" ] && continue; running=$(docker inspect --format \'{{.State.Running}}\' "$container" 2>/dev/null) '

View on GitHub (pinned to 70b9acc424)

Solutions

1. Check whether the ScheduledVolumeBackup and Server rows still exist; if the schedule was deleted but the server remains, restart the containers manually using the execution's stop_container_ids (`docker start <id> ...`)
2. If the server itself is gone, the containers were destroyed with it - nothing to recover
3. Clear stop_recovery_pending on the execution once containers are handled so the job stops retrying

// refuse to stop containers unless recovery is provably possible
$server = $backup->server();
if (! $server) {
    // skip the 'stop containers' option - cannot guarantee restart
    $backup->update(['stopContainers' => false]);
}

function recoveryTargetAvailable($execution): bool
{
    return $execution->scheduledVolumeBackup?->server() !== null;
}

When it happens

Trigger: The ScheduledVolumeBackup row or the Server row was deleted between the backup run and the recovery job, so the optional chain resolves to null; or the server() lookup returns null because the relation is broken.

Common situations: Deleting a server or its backup schedule while an execution still shows stop_recovery_pending; manual database edits removing rows mid-flight; recreating servers with fresh ids during a backup window.

Related errors

• The selected S3 storage no longer exists. S3 backup has been
• The storage backup archive is empty or was not created.
• Invalid Cron / Human expression
• Wait for the running storage backup and recovery operations
• The server is unavailable, so local backup archives cannot b

AI-assisted analysis of coollabsio/coolify@70b9acc424 (2026-08-17). Data as JSON: /api/errors/5023d30dae28820b. Report an issue: GitHub.