{"record":{"id":"5a2b8868d6d9f18e","repo":"phacility/phabricator","slug":"failed-to-reconnect-to-master-database-and-release","errorCode":null,"errorMessage":"Failed to reconnect to master database and release held write lock (\"%s\") on device \"%s\" for repository \"%s\" after trying for %s seconds(s). This repository will be frozen.","messagePattern":"Failed to reconnect to master database and release held write lock \\(\"(.+?)\"\\) on device \"(.+?)\" for repository \"(.+?)\" after trying for (.+?) seconds\\(s\\)\\. This repository will be frozen\\.","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"critical","filePath":"src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php","lineNumber":582,"sourceCode":"\n        $this->logLine(\n          pht(\n            'This process will spend %s more second(s) attempting to '.\n            'recover, then give up.',\n            new PhutilNumber($duration)));\n      }\n\n      sleep(1);\n    }\n\n    if ($did_release) {\n      if ($already_failed) {\n        $this->logLine(\n          pht('RECOVERED. Link to master database was restored.'));\n      }\n      $this->logLine(pht('Released cluster write lock.'));\n    } else {\n      throw new Exception(\n        pht(\n          'Failed to reconnect to master database and release held write '.\n          'lock (\"%s\") on device \"%s\" for repository \"%s\" after trying '.\n          'for %s seconds(s). This repository will be frozen.',\n          $this->clusterWriteOwner,\n          $device->getName(),\n          $this->getDisplayName(),\n          new PhutilNumber($duration)));\n    }\n\n    // We can continue even if we've lost this lock, everything is still\n    // consistent.\n    try {\n      $this->clusterWriteLock->unlock();\n    } catch (Exception $ex) {\n      // Ignore.\n    }\n","sourceCodeStart":564,"sourceCodeEnd":600,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php#L564-L600","documentation":"After a write, the engine must release the held cluster write lock by writing to the master database. It retried in a loop (the source shows sleep(1) iterations logging progress) but could not reconnect and release within the budget, so it throws and the repository stays frozen: the node cannot prove the lock was released, and Phabricator prefers frozen consistency over split-brain. The frozen state outlives the exception and needs operator cleanup.","triggerScenarios":"A network partition or master database outage lasting longer than the retry window occurs while this node holds a repository write lock after completing a push.","commonSituations":"Master DB failover or maintenance coinciding with pushes; network blips between repository nodes and the DB master; an overloaded master timing out writes.","solutions":["Restore master database connectivity first; that is the root cause","Once reconnected, follow the frozen-repository / 'Write Interruptions' recovery steps in the Cluster: Repositories documentation and re-run 'bin/repository update <monogram>' to confirm the repository unfreezes","Verify consistency of every device's copy before allowing new writes","Reduce recurrence with HA for the master DB and alerting on partition duration"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// pre-flight: verify master DB reachability before accepting writes on a cluster node\n$conn = id(new PhabricatorRepositoryWorkingCopyVersion())->establishConnection('w');\nqueryfx($conn, 'SELECT 1');","typeGuard":null,"tryCatchPattern":"try {\n  $engine->synchronizeWorkingCopyBeforeWrite();\n} catch (Exception $ex) {\n  // repository is FROZEN: page the operator, halt all writers on this repository, do not auto-unfreeze\n}","preventionTips":["Provide HA / redundant networking for the master database","Alert on repeated lock-release retries; they precede a freeze","Test DB failover drills during quiet periods, not push windows"],"tags":["cluster","database-outage","frozen-repository","write-lock","phabricator"],"backgroundTag":"database-unreachable","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}