coollabsio/coolify · error · Exception

Failed to acquire lock for SSH key: {$keyLocation}

Error message

Failed to acquire lock for SSH key: {$keyLocation}

What it means

Thrown by PrivateKey::storeInFileSystem() (app/Models/PrivateKey.php:222). After successfully fopen()ing the lock file, the code takes an exclusive flock(LOCK_EX) to prevent two workers from interleaving writes to the same key file. flock() returning false is rare on local filesystems and typically indicates the file lives on a filesystem that does not support advisory locks — most commonly NFS (especially NFSv4 without local locking), some FUSE mounts, or a Windows/SMB share.

Source

Thrown at app/Models/PrivateKey.php:222

    public function storeInFileSystem()
    {
        $filename = "ssh_key@{$this->uuid}";
        $disk = Storage::disk('ssh-keys');
        $keyLocation = $this->getKeyLocation();
        $lockFile = $keyLocation.'.lock';

        // Ensure the storage directory exists and is writable
        $this->ensureStorageDirectoryExists();

        // Use file locking to prevent concurrent writes from corrupting the key
        $lockHandle = fopen($lockFile, 'c');
        if ($lockHandle === false) {
            throw new \Exception("Failed to open lock file for SSH key: {$lockFile}");
        }

        try {
            if (! flock($lockHandle, LOCK_EX)) {
                throw new \Exception("Failed to acquire lock for SSH key: {$keyLocation}");
            }

            // Attempt to store the private key
            $success = $disk->put($filename, $this->private_key);

            if (! $success) {
                throw new \Exception("Failed to write SSH key to filesystem. Check disk space and permissions for: {$keyLocation}");
            }

            // Verify the file was actually created and has content
            if (! $disk->exists($filename)) {
                throw new \Exception("SSH key file was not created: {$keyLocation}");
            }

            $storedContent = $disk->get($filename);
            if (empty($storedContent) || $storedContent !== $this->private_key) {
                $disk->delete($filename); // Clean up the bad file
                throw new \Exception("SSH key file content verification failed: {$keyLocation}");

View on GitHub (pinned to 70b9acc424)

Solutions

  1. Move the ssh-keys storage off the network filesystem onto the container's local volume (local persistent dir for /data/coolify/ssh).
  2. If NFS must be used, mount it with locking enabled (nolock removed; vers=4 with a running lock manager).
  3. Retry the key save after remounting; the routine cleans up the lock handle in its finally block.
Defensive patterns

Strategy: fallback

Validate before calling

// Detect lock-incapable storage before relying on flock
$lockTest = $privateKey->getKeyLocation().'.probe.lock';
$h = @fopen($lockTest, 'c');
if ($h === false || ! @flock($h, LOCK_EX)) {
    // advisory locks unsupported — move ssh-keys storage to a local filesystem
}@fclose($h); @unlink($lockTest);

Try / catch

try {
    $privateKey->storeInFileSystem();
} catch (\Exception $e) {
    if (str_contains($e->getMessage(), 'Failed to acquire lock')) {
        // NFS/FUSE storage: remount with locking or relocate the ssh-keys disk, then retry
    }
    throw $e;
}

Prevention

When it happens

Trigger: Storing/updating a PrivateKey when the 'ssh-keys' disk root is backed by NFS/FUSE/SMB where flock(LOCK_EX) fails; the lock file opened fine but advisory locking is unsupported.

Common situations: Self-hosted Coolify with /data/coolify on an NFS export 'for convenience'; NAS-backed bind mounts; container filesystems backed by unusual storage drivers.

Related errors


AI-assisted analysis of coollabsio/coolify@70b9acc424 (2026-08-17). Data as JSON: /api/errors/0d22cd9d6917fdd8. Report an issue: GitHub.