{"record":{"id":"5023d30dae28820b","repo":"coollabsio/coolify","slug":"the-server-is-unavailable-for-container-recovery","errorCode":null,"errorMessage":"The server is unavailable for container recovery.","messagePattern":"The server is unavailable for container recovery\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"critical","filePath":"app/Jobs/VolumeBackupRecoveryJob.php","lineNumber":65,"sourceCode":"    public static function recover(ScheduledVolumeBackupExecution $execution): void\n    {\n        $execution->loadMissing('scheduledVolumeBackup.backupable.resource');\n\n        if ($execution->stop_recovery_pending) {\n            self::recoverContainers($execution);\n        }\n\n        if ($execution->s3_cleanup_pending) {\n            self::cleanupS3Upload($execution);\n        }\n    }\n\n    private static function recoverContainers(ScheduledVolumeBackupExecution $execution): void\n    {\n        $server = $execution->scheduledVolumeBackup?->server();\n\n        if (! $server) {\n            throw new \\RuntimeException('The server is unavailable for container recovery.');\n        }\n\n        $stateFile = self::stateFile($execution);\n        $output = instant_remote_process(\n            ['cat '.escapeshellarg($stateFile).' 2>/dev/null || true'],\n            $server,\n            disableMultiplexing: true,\n        );\n        $containers = collect(preg_split('/\\s+/', trim((string) $output)))\n            ->filter(fn (string $container): bool => preg_match('/^[a-f0-9]{6,64}$/i', $container) === 1)\n            ->values()\n            ->all();\n        $execution->update(['stop_container_ids' => $containers]);\n\n        $remainingFile = $stateFile.'.remaining';\n        $script = 'status=0; : > '.escapeshellarg($remainingFile).'; '\n            .'if [ -f '.escapeshellarg($stateFile).' ]; then while IFS= read -r container; do '\n            .'[ -z \"$container\" ] && continue; running=$(docker inspect --format \\'{{.State.Running}}\\' \"$container\" 2>/dev/null) '","sourceCodeStart":47,"sourceCodeEnd":83,"githubUrl":"https://github.com/coollabsio/coolify/blob/70b9acc42467278373e00de77abb40684e25b395/app/Jobs/VolumeBackupRecoveryJob.php#L47-L83","documentation":"After a volume backup that stopped containers (stop_recovery_pending), VolumeBackupRecoveryJob restarts them from the saved container ids. recoverContainers() resolves the server through `$execution->scheduledVolumeBackup?->server()`; if the scheduled backup or its server can no longer be resolved, it throws - the containers stopped for the backup remain stopped.","triggerScenarios":"The ScheduledVolumeBackup row or the Server row was deleted between the backup run and the recovery job, so the optional chain resolves to null; or the server() lookup returns null because the relation is broken.","commonSituations":"Deleting a server or its backup schedule while an execution still shows stop_recovery_pending; manual database edits removing rows mid-flight; recreating servers with fresh ids during a backup window.","solutions":["Check whether the ScheduledVolumeBackup and Server rows still exist; if the schedule was deleted but the server remains, restart the containers manually using the execution's stop_container_ids (`docker start <id> ...`)","If the server itself is gone, the containers were destroyed with it - nothing to recover","Clear stop_recovery_pending on the execution once containers are handled so the job stops retrying"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// refuse to stop containers unless recovery is provably possible\n$server = $backup->server();\nif (! $server) {\n    // skip the 'stop containers' option - cannot guarantee restart\n    $backup->update(['stopContainers' => false]);\n}","typeGuard":"function recoveryTargetAvailable($execution): bool\n{\n    return $execution->scheduledVolumeBackup?->server() !== null;\n}","tryCatchPattern":null,"preventionTips":["Never delete a server or backup schedule while any execution shows stop_recovery_pending","Keep stop_container_ids and the /tmp state file intact until recovery confirms success","Alert on executions stuck with stop_recovery_pending = true - those mean downtime"],"tags":["backup","container-recovery","dangling-reference","downtime","scheduled-task"],"backgroundTag":"dangling-foreign-key-reference","analyzedSha":"70b9acc42467278373e00de77abb40684e25b395","analyzedAt":"2026-08-17T01:41:01.313Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}