phacility/phabricator · error · PhutilProxyException

Elasticsearch server returned invalid JSON!

Error message

Elasticsearch server returned invalid JSON!

What it means

Thrown by PhabricatorElasticFulltextStorageEngine::executeRequest() when a GET request to a configured Elasticsearch host succeeds at the HTTP level but the response body fails phutil_json_decode(). The host is marked as having failed its health check (didHealthCheck(false)) and the PhutilJSONParserException is rewrapped in a PhutilProxyException. In practice it means the endpoint configured in cluster.search is not speaking the Elasticsearch HTTP API, so every engine operation (indexExists, executeSearch, getIndexStats, indexing writes) fails.

Source

Thrown at src/applications/search/fulltextstorage/PhabricatorElasticFulltextStorageEngine.php:536

      list($body) = $future->resolvex();
    } catch (HTTPFutureResponseStatus $ex) {
      if ($ex->isTimeout() || (int)$ex->getStatusCode() > 499) {
        $host->didHealthCheck(false);
      }
      throw $ex;
    }

    if ($method != 'GET') {
      return null;
    }

    try {
      $data = phutil_json_decode($body);
      $host->didHealthCheck(true);
      return $data;
    } catch (PhutilJSONParserException $ex) {
      $host->didHealthCheck(false);
      throw new PhutilProxyException(
        pht('Elasticsearch server returned invalid JSON!'),
        $ex);
    }

  }

}

View on GitHub (pinned to 5720a38cfe)

Solutions

1. From the Phabricator host, run curl -v on the exact configured scheme://host:port/ and inspect the raw body - if it is not JSON you have found the problem
2. Fix the cluster.search config so each host entry points at the Elasticsearch HTTP endpoint (correct host, port, protocol)
3. Whitelist the Phabricator host in any proxy/auth layer in front of ES, or remove the layer from that port
4. Confirm the Elasticsearch major version is one this Phabricator release supports (unsupported versions often serve unexpected payloads)
5. Re-run bin/search status after each change; the host must report healthy before reindexing

Example fix

// before (conf/local.json): port 80 is the web server, not Elasticsearch
"cluster.search": [{"type": "elasticsearch", "hosts": [{"host": "127.0.0.1", "port": 80, "protocol": "http"}]}]

// after: point at the real Elasticsearch HTTP endpoint
"cluster.search": [{"type": "elasticsearch", "hosts": [{"host": "127.0.0.1", "port": 9200, "protocol": "http"}]}]

// Preflight the configured host exactly the way the engine does, before relying on it
$uri = "{$protocol}://{$host}:{$port}/";
$body = @file_get_contents($uri);
if ($body === false || json_decode($body) === null) {
  // host is not speaking JSON: fix cluster.search before indexing
}

try {
  $results = $engine->executeSearch($query);
} catch (PhutilProxyException $ex) {
  if ($ex->getPrevious() instanceof PhutilJSONParserException) {
    // Host answered non-JSON: mark backend down, alert, do not retry blindly
  }
  throw $ex;
}

When it happens

Trigger: Any bin/search command or object-edit indexing against a cluster.search host whose HTTP response is not JSON: pointing at a port served by a web server or reverse proxy that returns an HTML error/login page, an auth proxy (e.g. basic auth or shield/x-pack) returning a non-JSON body, a firewall/appliance interception page, or a truncated/garbled response.

Common situations: cluster.search host/port typo (e.g. pointing at the Phabricator web host instead of ES 9200), ES behind an SSL terminator or SSO proxy, docker port mapping exposing the wrong service, ES not running and something else answering on that port, network middleboxes injecting error pages.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Search cluster has no hosts for role "%s".
• All of the configured Fulltext Search services failed.
• Failed to rebuild search index for any documents.
• Invalid search engine type: %s. Valid types are: %s.
• Writes to search services failed while reindexing document "

AI-assisted analysis of phacility/phabricator@5720a38cfe (2026-08-21). Data as JSON: /api/errors/b23918fd3c1f740f. Report an issue: GitHub.