{"record":{"id":"d76b6bcec936c842","repo":"phacility/phabricator","slug":"unable-to-establish-a-connection-to-any-database-h-d76b6b","errorCode":null,"errorMessage":"Unable to establish a connection to any database host (while trying \"%s\"). All masters and replicas are completely unreachable.","messagePattern":"Unable to establish a connection to any database host \\(while trying \"(.+?)\"\\)\\. All masters and replicas are completely unreachable\\.","errorType":"exception","errorClass":"PhabricatorClusterStrandedException","httpStatus":null,"severity":"critical","filePath":"src/infrastructure/storage/lisk/PhabricatorLiskDAO.php","lineNumber":177,"sourceCode":"        $database));\n  }\n\n  private function raiseUnreachable($database, Exception $proxy = null) {\n    $message = pht(\n      'Unable to establish a connection to any database host '.\n      '(while trying \"%s\"). All masters and replicas are completely '.\n      'unreachable.',\n      $database);\n\n    if ($proxy) {\n      $proxy_message = pht(\n        '%s: %s',\n        get_class($proxy),\n        $proxy->getMessage());\n      $message = $message.\"\\n\\n\".$proxy_message;\n    }\n\n    throw new PhabricatorClusterStrandedException($message);\n  }\n\n\n  /**\n   * @task config\n   */\n  public function getTableName() {\n    $str = 'phabricator';\n    $len = strlen($str);\n\n    $class = strtolower(get_class($this));\n    if (!strncmp($class, $str, $len)) {\n      $class = substr($class, $len);\n    }\n    $app = $this->getApplicationName();\n    if (!strncmp($class, $app, strlen($app))) {\n      $class = substr($class, strlen($app));\n    }","sourceCodeStart":159,"sourceCodeEnd":195,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php#L159-L195","documentation":"Thrown by PhabricatorLiskDAO::raiseUnreachable() (src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:177) as PhabricatorClusterStrandedException after the connection loop failed against every configured host. The DAO tried each master and each replica in the cluster configuration and none of them accepted a TCP/MySQL connection, so Phabricator considers itself 'stranded' with no database backend at all. When a previous exception is available it is appended to the message (class name plus the underlying MySQL/protocol error), which is the real root cause to read first.","triggerScenarios":"Any Lisk DAO operation that opens a live connection (query, write, or establishing a connection inside a web request or bin/storage workflow) while every host in the configured database cluster is down: mysqld stopped on all servers, wrong host/port in the cluster config, firewall or network partition blocking 3306 from all web/CLI hosts, or every host refusing auth (auth failure also counts as unreachable). It is raised from establishLiveConnection() when both $master_exception and the replica attempts are exhausted; if no hosts are configured at all, raiseUnconfigured() is thrown instead.","commonSituations":"MySQL service not started after a reboot; migration or container restart where the DB container comes up after the app; typo'd db.host/db.port or a changed MySQL bind-address; firewall/SELinux blocking the port; all cluster hosts failing credentials after a password rotation; DNS resolving the DB hostname to a dead IP.","solutions":["Read the appended proxy exception in the message — it names the underlying cause (connection refused, timeout, auth denied) for the last host tried.","Verify mysqld is running and reachable from this host: service mysql status, then mysql -h <host> -P <port> -u <user> -p using the exact credentials from the Phabricator config.","Check network path: telnet <host> 3306 / nc -vz <host> 3306; open the firewall or fix DNS if it fails.","Verify the cluster configuration (database hosts/ref config, port, user, pass) matches reality: ./bin/config get mysql.configuration or the relevant db.* keys; correct with ./bin/config set.","If auth is the failure, fix grants/credentials on the MySQL side (or in config), then re-run the failing command.","If hosts were intentionally removed, ensure at least one reachable master or replica remains configured."],"exampleFix":"# before: all hosts down / misconfigured\nmysql -h10.0.0.5 -P3306 -u phabricator -p   # ERROR 2003: Can't connect\n\n# after: mysqld running, reachable, credentials valid\nsudo systemctl start mysql\nmysql -h10.0.0.5 -P3306 -u phabricator -p   # connects\n./bin/storage status","handlingStrategy":"fallback","validationCode":"// Pre-flight: verify at least one configured DB host accepts a TCP connection\n// before running an operation that will strand on failure.\nfunction anyDatabaseHostReachable(array $hosts, $timeout = 2) {\n  foreach ($hosts as $host => $port) {\n    $sock = @fsockopen($host, $port, $errno, $errstr, $timeout);\n    if ($sock) { fclose($sock); return true; }\n  }\n  return false;\n}\n// $hosts from your cluster config, e.g. array('10.0.0.5' => 3306);","typeGuard":"// Narrow a caught exception before deciding it is a stranding failure.\nfunction isStrandedException(Exception $ex) {\n  return ($ex instanceof PhabricatorClusterStrandedException);\n}","tryCatchPattern":"try {\n  $dao->establishLiveConnection('r');\n} catch (PhabricatorClusterStrandedException $ex) {\n  // Read the appended proxy message for the real cause (refused/timeout/auth).\n  phlog($ex->getMessage());\n  // Serve a degraded 'database offline' page; do NOT loop-retry immediately.\n  throw new AphrontRetryExceptionHandler($request, $ex); // let web stack back off\n}","preventionTips":["Monitor mysqld on every configured master/replica and alert before the last healthy host dies.","Keep at least two database refs (one master, one replica) configured so a single host outage does not strand Phabricator.","Include a TCP reachability check (fsockopen to host:port) in deployment/pre-flight scripts.","Watch daemon/web logs for the proxy exception text appended to this error; it carries the underlying MySQL error."],"tags":["phabricator","mysql","database","cluster","connection-refused","lisk-dao"],"backgroundTag":"database-unreachable","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}