phacility/phabricator · critical · PhutilAggregateException

All Fulltext Search hosts failed:

Error message

All Fulltext Search hosts failed:

What it means

executeSearch() in the Elasticsearch fulltext engine loops over every configured search host with the 'read' role and tries the query on each; only if ALL of them throw does it rethrow as PhutilAggregateException 'All Fulltext Search hosts failed:' wrapping every per-host exception. The aggregate carries the real causes (connection refused, 4xx/5xx HTTP status, malformed index) in its ->getExceptions() (displayed in the message) - those inner exceptions, not the wrapper, are what to debug.

Source

Thrown at src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:262

    // Don't use '/_search' for the case that there is something
    // else in the index (for example if 'phabricator' is only an alias to
    // some bigger index). Use '/$types/_search' instead.
    $uri = '/'.implode(',', $types).'/_search';

    $spec = $this->buildSpec($query);
    $exceptions = array();

    foreach ($this->service->getAllHostsForRole('read') as $host) {
      try {
        $response = $this->executeRequest($host, $uri, $spec);
        $phids = ipull($response['hits']['hits'], '_id');
        return $phids;
      } catch (Exception $e) {
        $exceptions[] = $e;
      }
    }
    throw new PhutilAggregateException(pht('All Fulltext Search hosts failed:'),
      $exceptions);
  }

  public function indexExists(PhabricatorElasticsearchHost $host = null) {
    if (!$host) {
      $host = $this->getHostForRead();
    }
    try {
      if ($this->version >= 5) {
        $uri = '/_stats/';
        $res = $this->executeRequest($host, $uri, array());
        return isset($res['indices']['phabricator']);
      } else if ($this->version >= 2) {
        $uri = '';
      } else {
        $uri = '/_status/';
      }
      return (bool)$this->executeRequest($host, $uri, array());

View on GitHub (pinned to 5720a38cfe)

Solutions

1. Read the wrapped exceptions: each line after 'All Fulltext Search hosts failed:' names the per-host failure (Connection refused, 404, 400 mapping error) - fix that root cause first.
2. Check ES health: curl http://<es-host>:9200/_cluster/health and confirm the 'phabricator' index exists; if absent, trigger reindex (bin/search index, or manage search with ./bin/search) and verify indexExists().
3. Verify the search cluster config: `./bin/config get cluster.search-service-config` - host, port, port (9200 vs 9300), protocol, and that the host role includes 'read'.
4. If ES was upgraded, match Phabricator's configured version to the real one and rebuild the index per the upgrade docs; as a stopgap, switch search back to MySQL/NULL engine so the UI keeps working while you repair ES.

Example fix

// before: single-host config that has no fallback
{
  "cluster.search-service-config": [{
    "type": "elasticsearch",
    "hosts": [{ "host": "es1.example.com", "port": 9200, "protocol": "http", "roles": { "read": true, "write": true } }]
  }]
}

// after: two read replicas so one dead host does not kill search
{
  "cluster.search-service-config": [{
    "type": "elasticsearch",
    "hosts": [
      { "host": "es1.example.com", "port": 9200, "protocol": "http", "roles": { "read": true, "write": true } },
      { "host": "es2.example.com", "port": 9200, "protocol": "http", "roles": { "read": true } }
    ]
  }]
}

// Health-check the search cluster before relying on it
foreach ($service->getAllHostsForRole('read') as $host) {
  if (!$engine->indexExists($host)) { /* mark degraded, alert ops */ }
}

try {
  $phids = $engine->executeSearch($query);
} catch (PhutilAggregateException $ex) {
  foreach ($ex->getExceptions() as $inner) {
    phlog('search host failure: '.$inner->getMessage()); // real causes live here
  }
  // degrade gracefully: fall back to no fulltext / cached results
  return array();
}

When it happens

Trigger: Any fulltext search (global search, Maniphest/Differential query with a text term) when: every ES host in cluster.search-service-config is down/unreachable; the 'phabricator' index does not exist (never built) and ES returns 404 for /_search; ES version mismatch producing HTTP errors; authentication/proxy rejecting requests; index mapping broken after a major ES upgrade.

Common situations: Elasticsearch service stopped or crashed (daemon restart, OOM); firewall/port change after infra work; Phabricator pointing at the wrong host/port in the cluster config; first run after enabling ES search before any documents were indexed (missing index); ES major-version upgrades (1.x -> 2.x -> 5.x+) that change API endpoints Phabricator uses.

Related errors

• Query offset is too large. offset+limit=%s (max=%s)
• Unable to allocate any binding as a resource.
• Parameter "fullText" is no longer supported. Use method "man
• Invalid search engine type: %s. Valid types are: %s.
• Search cluster has no hosts for role "%s".

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/952110f9973e27bd. Report an issue: GitHub.