phacility/phabricator · critical · Exception

Failed to reconnect to master database and release held writ

Error message

Failed to reconnect to master database and release held write lock ("%s") on device "%s" for repository "%s" after trying for %s seconds(s). This repository will be frozen.

What it means

After a write, the engine must release the held cluster write lock by writing to the master database. It retried in a loop (the source shows sleep(1) iterations logging progress) but could not reconnect and release within the budget, so it throws and the repository stays frozen: the node cannot prove the lock was released, and Phabricator prefers frozen consistency over split-brain. The frozen state outlives the exception and needs operator cleanup.

Source

Thrown at src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php:582

        $this->logLine(
          pht(
            'This process will spend %s more second(s) attempting to '.
            'recover, then give up.',
            new PhutilNumber($duration)));
      }

      sleep(1);
    }

    if ($did_release) {
      if ($already_failed) {
        $this->logLine(
          pht('RECOVERED. Link to master database was restored.'));
      }
      $this->logLine(pht('Released cluster write lock.'));
    } else {
      throw new Exception(
        pht(
          'Failed to reconnect to master database and release held write '.
          'lock ("%s") on device "%s" for repository "%s" after trying '.
          'for %s seconds(s). This repository will be frozen.',
          $this->clusterWriteOwner,
          $device->getName(),
          $this->getDisplayName(),
          new PhutilNumber($duration)));
    }

    // We can continue even if we've lost this lock, everything is still
    // consistent.
    try {
      $this->clusterWriteLock->unlock();
    } catch (Exception $ex) {
      // Ignore.
    }

View on GitHub (pinned to 5720a38cfe)

Solutions

1. Restore master database connectivity first; that is the root cause
2. Once reconnected, follow the frozen-repository / 'Write Interruptions' recovery steps in the Cluster: Repositories documentation and re-run 'bin/repository update <monogram>' to confirm the repository unfreezes
3. Verify consistency of every device's copy before allowing new writes
4. Reduce recurrence with HA for the master DB and alerting on partition duration

// pre-flight: verify master DB reachability before accepting writes on a cluster node
$conn = id(new PhabricatorRepositoryWorkingCopyVersion())->establishConnection('w');
queryfx($conn, 'SELECT 1');

try {
  $engine->synchronizeWorkingCopyBeforeWrite();
} catch (Exception $ex) {
  // repository is FROZEN: page the operator, halt all writers on this repository, do not auto-unfreeze
}

When it happens

Trigger: A network partition or master database outage lasting longer than the retry window occurs while this node holds a repository write lock after completing a push.

Common situations: Master DB failover or maintenance coinciding with pushes; network blips between repository nodes and the DB master; an overloaded master timing out writes.

Related errors

• This host has device ID "%s", but there is no corresponding
• Configuration file specifies cluster peer ("%s", at index "%
• Configuration file specifies cluster peer "%s" more than onc
• Failed to load repository cluster service.
• Failed to acquire read lock after waiting %s second(s). You

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/5a2b8868d6d9f18e. Report an issue: GitHub.