{"record":{"id":"1450472e61826943","repo":"openzipkin/zipkin","slug":"no-content-reading-cluster-health","errorCode":null,"errorMessage":"No content reading cluster health","messagePattern":"No content reading cluster health","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"zipkin-storage/elasticsearch/src/main/java/zipkin2/elasticsearch/ElasticsearchStorage.java","lineNumber":296,"sourceCode":"\n  /** This is blocking so that we can determine if the cluster is healthy or not */\n  @Override public CheckResult check() {\n    return ensureIndexTemplatesAndClusterReady(indexNameFormatter().formatType(TYPE_SPAN));\n  }\n\n  /**\n   * This allows the health check to display problems, such as access, installing the index\n   * template. It also helps reduce traffic sent to nodes still initializing (when guarded on the\n   * check result). Finally, this reads the cluster health of the index as it can go down after the\n   * one-time initialization passes.\n   */\n  CheckResult ensureIndexTemplatesAndClusterReady(String index) {\n    try {\n      version(); // ensure the version is available (even if we already cached it)\n      ensureIndexTemplates(); // called only once, so we have to double-check health\n      AggregatedHttpRequest request = AggregatedHttpRequest.of(GET, \"/_cluster/health/\" + index);\n      CheckResult result = http().newCall(request, READ_STATUS, \"get-cluster-health\").execute();\n      if (result == null) throw new IllegalArgumentException(\"No content reading cluster health\");\n      return result;\n    } catch (Throwable e) {\n      Call.propagateIfFatal(e);\n      // Wrapping interferes with humans intended to read this message:\n      //\n      // Unwrap the marker exception as the health check is not relevant for the throttle component.\n      // Unwrap any IOException from the first call to ensureIndexTemplates()\n      if (e instanceof RejectedExecutionException || e instanceof UncheckedIOException) {\n        return CheckResult.failed(e.getCause());\n      }\n      return CheckResult.failed(e);\n    }\n  }\n\n  volatile boolean ensuredTemplates;\n\n  // synchronized since we don't want overlapping calls to apply the index templates\n  void ensureIndexTemplates() {","sourceCodeStart":278,"sourceCodeEnd":314,"githubUrl":"https://github.com/openzipkin/zipkin/blob/878ce2a1fad54ca941d17fdcf2e1d924b148eb1f/zipkin-storage/elasticsearch/src/main/java/zipkin2/elasticsearch/ElasticsearchStorage.java#L278-L314","documentation":"ElasticsearchStorage.ensureIndexTemplatesAndClusterReady(index) performs the one-time index-template installation and then issues GET /_cluster/health/<index> through the READ_STATUS body converter. The converter is expected to always produce a CheckResult; result == null means the HTTP call completed but no content/body was converted — i.e. the cluster-health response was empty — and that is surfaced as IllegalArgumentException instead of a silent pass.","triggerScenarios":"The health-check path (CheckResult check() / first query after startup) when /_cluster/health/<zipkin*> returns an empty body: e.g. an intermediate proxy or load balancer answering 200 with zero bytes, a misrouted host, or a node closing the response early.","commonSituations":"Elasticsearch behind nginx/HAProxy/Kubernetes ingress that strips or buffers responses; connecting through a service mesh; a URL with a wrong port hitting a non-ES service.","solutions":["curl -v the exact URL Zipkin uses (http://<host>:9200/_cluster/health/zipkin-span-<date>) and confirm a JSON body with a status field is returned.","Fix the proxy/ingress in front of Elasticsearch so it passes the response body through unchanged.","Verify ES_HOSTS points at real Elasticsearch/OpenSearch HTTP endpoints (port 9200, not the transport port).","The exception is wrapped into CheckResult.failed by the surrounding catch, so the message appears in the health endpoint — use it to diagnose rather than crash."],"exampleFix":"# before: proxy returns empty 200 for /_cluster/health/*\n# -> IllegalArgumentException: No content reading cluster health\n\n# after: verify direct node response and bypass/fix the proxy\n$ curl -s http://es:9200/_cluster/health/zipkin-span-2026-08-14\n{\"cluster_name\":\"es\",\"status\":\"green\",...}","handlingStrategy":"retry","validationCode":"// smoke-test the exact health URL before/at startup\ntry (var client = HttpClient.newHttpClient()) {\n  var resp = client.send(HttpRequest.newBuilder(URI.create(esHost + \"/_cluster/health/zipkin-span\")).GET().build(),\n      HttpResponse.BodyHandlers.ofString());\n  if (resp.statusCode() != 200 || resp.body().isEmpty()) {\n    throw new IllegalStateException(\"Proxy/node returned empty cluster health body\");\n  }\n}","typeGuard":null,"tryCatchPattern":"// health checks return CheckResult instead of throwing; inspect it\nCheckResult health = storage.check();\nif (!health.ok()) {\n  // log health.error(); retry or alert — this error appears as its message\n}","preventionTips":["Smoke-test /_cluster/health/<index> through the exact proxy path Zipkin uses before deploying.","Avoid LB/proxy rules that rewrite or empty ES responses (especially 200-with-empty-body).","Treat check() failures as alerts, not crashes; the underlying cause is in CheckResult.error()."],"tags":["elasticsearch","health-check","proxy","cluster-health","storage"],"backgroundTag":null,"analyzedSha":"878ce2a1fad54ca941d17fdcf2e1d924b148eb1f","analyzedAt":"2026-08-14T15:17:09.895Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}