What causes "Search cluster has no hosts for role "%s"." in phacility/phabricator?

Inspect `getAllHostsForRole($role)`: an empty array means config — fix the `roles` maps or remove the service-level role disable.

phacility/phabricator · error · PhabricatorClusterNoHostForRoleException

Search cluster has no hosts for role "%s".

Error message

Search cluster has no hosts for role "%s".

What it means

PhabricatorSearchService::getAnyHostForRole($role) picks a random host with the requested role (`read` or `write`) that is currently passing its health checks. It throws PhabricatorClusterNoHostForRoleException when no host has the role at all — including when the role is disabled service-wide via `roles => array('read' => false)` or a top-level `'read' => false` — or when every matching host is temporarily unhealthy after recent connection failures.

Source

Thrown at src/infrastructure/cluster/search/PhabricatorSearchService.php:129

  }


  /**
   * Get a random host reference with the specified role, skipping hosts which
   * failed recent health checks.
   * @throws PhabricatorClusterNoHostForRoleException if no healthy hosts match.
   * @return PhabricatorSearchHost
   */
  public function getAnyHostForRole($role) {
    $hosts = $this->getAllHostsForRole($role);
    shuffle($hosts);
    foreach ($hosts as $host) {
      $health = $host->getHealthRecord();
      if ($health->getIsHealthy()) {
        return $host;
      }
    }
    throw new PhabricatorClusterNoHostForRoleException($role);
  }


  /**
   * Get all configured hosts for this service which have the specified role.
   * @return PhabricatorSearchHost[]
   */
  public function getAllHostsForRole($role) {
    // if the role is explicitly set to false at the top level, then all hosts
    // have the role disabled.
    if (idx($this->config, $role) === false) {
      return array();
    }

    $hosts = array();
    foreach ($this->hosts as $host) {
      if ($host->hasRole($role)) {
        $hosts[] = $host;

View on GitHub (pinned to 5720a38cfe)

Solutions

Inspect `getAllHostsForRole($role)`: an empty array means config — fix the `roles` maps or remove the service-level role disable.
If hosts exist but all are unhealthy, restore the backend (service, network, credentials) and let automatic health checks mark them healthy again; sick hosts are retried as their health record recovers.
Check `bin/search status` / the Search Servers console for per-host health and failure counts.
As a stopgap, configure a writable MySQL fulltext service so reads can be served while the primary backend recovers.

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check before requesting a host:
$hosts = $service->getAllHostsForRole($role);
if (!$hosts) {
  // configuration problem: no host has this role (or role disabled at top level)
  return handleNoHostConfigured($role);
}
$healthy = array_filter($hosts, function ($h) {
  return $h->getHealthRecord()->getIsHealthy();
});
if (!$healthy) {
  // all sick: backend outage, health checks will recover them
  return deferOrFallback($role);
}

Try / catch

try {
  $host = $service->getAnyHostForRole('read');
} catch (PhabricatorClusterNoHostForRoleException $ex) {
  if (!$service->getAllHostsForRole('read')) {
    // permanent: fix cluster.search roles config
    throw $ex;
  }
  // transient: all hosts unhealthy; bounded backoff, then surface
  usleep(250000); // simple single retry backoff, do not loop unbounded
  $host = $service->getAnyHostForRole('read');
}

Prevention

Configure at least two hosts per role across services so one sick host cannot starve a role.
Monitor `bin/search status` and alert when health failure counts rise, before the last host flips unhealthy.
Never set `roles => array('read' => false)` on every host unless code paths requesting that role are also disabled.

When it happens

Trigger: Calling `getAnyHostForRole('write')` when all hosts set `roles => array('write' => false)`; the service config disables the role at the top level (`idx($this->config, $role) === false`); or the Elasticsearch/MySQL backend is down and health records have marked every host sick.

Common situations: Search backend outage while users run queries; read replicas marked non-readable during maintenance but code still requests reads; querying immediately after a network blip before health scores decay back to healthy.

Related errors

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/359f5c2a49b4f759. Report an issue: GitHub.