{"record":{"id":"952110f9973e27bd","repo":"phacility/phabricator","slug":"all-fulltext-search-hosts-failed","errorCode":null,"errorMessage":"All Fulltext Search hosts failed:","messagePattern":"All Fulltext Search hosts failed:","errorType":"exception","errorClass":"PhutilAggregateException","httpStatus":null,"severity":"critical","filePath":"src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php","lineNumber":262,"sourceCode":"\n    // Don't use '/_search' for the case that there is something\n    // else in the index (for example if 'phabricator' is only an alias to\n    // some bigger index). Use '/$types/_search' instead.\n    $uri = '/'.implode(',', $types).'/_search';\n\n    $spec = $this->buildSpec($query);\n    $exceptions = array();\n\n    foreach ($this->service->getAllHostsForRole('read') as $host) {\n      try {\n        $response = $this->executeRequest($host, $uri, $spec);\n        $phids = ipull($response['hits']['hits'], '_id');\n        return $phids;\n      } catch (Exception $e) {\n        $exceptions[] = $e;\n      }\n    }\n    throw new PhutilAggregateException(pht('All Fulltext Search hosts failed:'),\n      $exceptions);\n  }\n\n  public function indexExists(PhabricatorElasticsearchHost $host = null) {\n    if (!$host) {\n      $host = $this->getHostForRead();\n    }\n    try {\n      if ($this->version >= 5) {\n        $uri = '/_stats/';\n        $res = $this->executeRequest($host, $uri, array());\n        return isset($res['indices']['phabricator']);\n      } else if ($this->version >= 2) {\n        $uri = '';\n      } else {\n        $uri = '/_status/';\n      }\n      return (bool)$this->executeRequest($host, $uri, array());","sourceCodeStart":244,"sourceCodeEnd":280,"githubUrl":"https://github.com/phacility/phabricator/blob/5720a38cfe95b00ca4be5016dd0d2f3195f4fa04/src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php#L244-L280","documentation":"executeSearch() in the Elasticsearch fulltext engine loops over every configured search host with the 'read' role and tries the query on each; only if ALL of them throw does it rethrow as PhutilAggregateException 'All Fulltext Search hosts failed:' wrapping every per-host exception. The aggregate carries the real causes (connection refused, 4xx/5xx HTTP status, malformed index) in its ->getExceptions() (displayed in the message) - those inner exceptions, not the wrapper, are what to debug.","triggerScenarios":"Any fulltext search (global search, Maniphest/Differential query with a text term) when: every ES host in cluster.search-service-config is down/unreachable; the 'phabricator' index does not exist (never built) and ES returns 404 for /_search; ES version mismatch producing HTTP errors; authentication/proxy rejecting requests; index mapping broken after a major ES upgrade.","commonSituations":"Elasticsearch service stopped or crashed (daemon restart, OOM); firewall/port change after infra work; Phabricator pointing at the wrong host/port in the cluster config; first run after enabling ES search before any documents were indexed (missing index); ES major-version upgrades (1.x -> 2.x -> 5.x+) that change API endpoints Phabricator uses.","solutions":["Read the wrapped exceptions: each line after 'All Fulltext Search hosts failed:' names the per-host failure (Connection refused, 404, 400 mapping error) - fix that root cause first.","Check ES health: curl http://<es-host>:9200/_cluster/health and confirm the 'phabricator' index exists; if absent, trigger reindex (bin/search index, or manage search with ./bin/search) and verify indexExists().","Verify the search cluster config: `./bin/config get cluster.search-service-config` - host, port, port (9200 vs 9300), protocol, and that the host role includes 'read'.","If ES was upgraded, match Phabricator's configured version to the real one and rebuild the index per the upgrade docs; as a stopgap, switch search back to MySQL/NULL engine so the UI keeps working while you repair ES."],"exampleFix":"// before: single-host config that has no fallback\n{\n  \"cluster.search-service-config\": [{\n    \"type\": \"elasticsearch\",\n    \"hosts\": [{ \"host\": \"es1.example.com\", \"port\": 9200, \"protocol\": \"http\", \"roles\": { \"read\": true, \"write\": true } }]\n  }]\n}\n\n// after: two read replicas so one dead host does not kill search\n{\n  \"cluster.search-service-config\": [{\n    \"type\": \"elasticsearch\",\n    \"hosts\": [\n      { \"host\": \"es1.example.com\", \"port\": 9200, \"protocol\": \"http\", \"roles\": { \"read\": true, \"write\": true } },\n      { \"host\": \"es2.example.com\", \"port\": 9200, \"protocol\": \"http\", \"roles\": { \"read\": true } }\n    ]\n  }]\n}","handlingStrategy":"retry","validationCode":"// Health-check the search cluster before relying on it\nforeach ($service->getAllHostsForRole('read') as $host) {\n  if (!$engine->indexExists($host)) { /* mark degraded, alert ops */ }\n}","typeGuard":null,"tryCatchPattern":"try {\n  $phids = $engine->executeSearch($query);\n} catch (PhutilAggregateException $ex) {\n  foreach ($ex->getExceptions() as $inner) {\n    phlog('search host failure: '.$inner->getMessage()); // real causes live here\n  }\n  // degrade gracefully: fall back to no fulltext / cached results\n  return array();\n}","preventionTips":["Monitor Elasticsearch with /_cluster/health and alert before it goes red","Configure at least two read-replica search hosts so one outage does not kill search","Run ./bin/search index after enabling or upgrading the search cluster","Pin and test the ES major version against Phabricator's supported range before upgrading","Keep MySQL as a fallback search engine in staging to compare behavior during outages"],"tags":["phabricator","elasticsearch","fulltext-search","service-unavailable","aggregate-exception"],"backgroundTag":"search-backend-unavailable","analyzedSha":"5720a38cfe95b00ca4be5016dd0d2f3195f4fa04","analyzedAt":"2026-08-21T05:07:25.672Z","schemaVersion":2},"datasetVersion":"2026-08-21T11:28:35.574Z"}