phacility/phabricator · critical · PhabricatorClusterStrandedException

Unable to establish a connection to any database host (while

Error message

Unable to establish a connection to any database host (while trying "%s"). All masters and replicas are completely unreachable.

What it means

Thrown by PhabricatorLiskDAO::raiseUnreachable() (src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:177) as PhabricatorClusterStrandedException after the connection loop failed against every configured host. The DAO tried each master and each replica in the cluster configuration and none of them accepted a TCP/MySQL connection, so Phabricator considers itself 'stranded' with no database backend at all. When a previous exception is available it is appended to the message (class name plus the underlying MySQL/protocol error), which is the real root cause to read first.

Source

Thrown at src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:177

        $database));
  }

  private function raiseUnreachable($database, Exception $proxy = null) {
    $message = pht(
      'Unable to establish a connection to any database host '.
      '(while trying "%s"). All masters and replicas are completely '.
      'unreachable.',
      $database);

    if ($proxy) {
      $proxy_message = pht(
        '%s: %s',
        get_class($proxy),
        $proxy->getMessage());
      $message = $message."\n\n".$proxy_message;
    }

    throw new PhabricatorClusterStrandedException($message);
  }


  /**
   * @task config
   */
  public function getTableName() {
    $str = 'phabricator';
    $len = strlen($str);

    $class = strtolower(get_class($this));
    if (!strncmp($class, $str, $len)) {
      $class = substr($class, $len);
    }
    $app = $this->getApplicationName();
    if (!strncmp($class, $app, strlen($app))) {
      $class = substr($class, strlen($app));
    }

View on GitHub (pinned to 5720a38cfe)

Solutions

Read the appended proxy exception in the message — it names the underlying cause (connection refused, timeout, auth denied) for the last host tried.
Verify mysqld is running and reachable from this host: service mysql status, then mysql -h <host> -P <port> -u <user> -p using the exact credentials from the Phabricator config.
Check network path: telnet <host> 3306 / nc -vz <host> 3306; open the firewall or fix DNS if it fails.
Verify the cluster configuration (database hosts/ref config, port, user, pass) matches reality: ./bin/config get mysql.configuration or the relevant db.* keys; correct with ./bin/config set.
If auth is the failure, fix grants/credentials on the MySQL side (or in config), then re-run the failing command.
If hosts were intentionally removed, ensure at least one reachable master or replica remains configured.

Example fix

# before: all hosts down / misconfigured
mysql -h10.0.0.5 -P3306 -u phabricator -p   # ERROR 2003: Can't connect

# after: mysqld running, reachable, credentials valid
sudo systemctl start mysql
mysql -h10.0.0.5 -P3306 -u phabricator -p   # connects
./bin/storage status

Defensive patterns

Strategy: fallback

Validate before calling

// Pre-flight: verify at least one configured DB host accepts a TCP connection
// before running an operation that will strand on failure.
function anyDatabaseHostReachable(array $hosts, $timeout = 2) {
  foreach ($hosts as $host => $port) {
    $sock = @fsockopen($host, $port, $errno, $errstr, $timeout);
    if ($sock) { fclose($sock); return true; }
  }
  return false;
}
// $hosts from your cluster config, e.g. array('10.0.0.5' => 3306);

Type guard

// Narrow a caught exception before deciding it is a stranding failure.
function isStrandedException(Exception $ex) {
  return ($ex instanceof PhabricatorClusterStrandedException);
}

Try / catch

try {
  $dao->establishLiveConnection('r');
} catch (PhabricatorClusterStrandedException $ex) {
  // Read the appended proxy message for the real cause (refused/timeout/auth).
  phlog($ex->getMessage());
  // Serve a degraded 'database offline' page; do NOT loop-retry immediately.
  throw new AphrontRetryExceptionHandler($request, $ex); // let web stack back off
}

Prevention

Monitor mysqld on every configured master/replica and alert before the last healthy host dies.
Keep at least two database refs (one master, one replica) configured so a single host outage does not strand Phabricator.
Include a TCP reachability check (fsockopen to host:port) in deployment/pre-flight scripts.
Watch daemon/web logs for the proxy exception text appended to this error; it carries the underlying MySQL error.

When it happens

Trigger: Any Lisk DAO operation that opens a live connection (query, write, or establishing a connection inside a web request or bin/storage workflow) while every host in the configured database cluster is down: mysqld stopped on all servers, wrong host/port in the cluster config, firewall or network partition blocking 3306 from all web/CLI hosts, or every host refusing auth (auth failure also counts as unreachable). It is raised from establishLiveConnection() when both $master_exception and the replica attempts are exhausted; if no hosts are configured at all, raiseUnconfigured() is thrown instead.

Common situations: MySQL service not started after a reboot; migration or container restart where the DB container comes up after the app; typo'd db.host/db.port or a changed MySQL bind-address; firewall/SELinux blocking the port; all cluster hosts failing credentials after a password rotation; DNS resolving the DB hostname to a dead IP.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/d76b6bcec936c842. Report an issue: GitHub.