What causes "Elasticsearch cluster died" in elastic/elasticsearch?

Read the node log that was being streamed (printed just above the exception) — the actual JVM exit cause is the last lines of that log.

elastic/elasticsearch · critical · GradleException

Elasticsearch cluster died

Error message

Elasticsearch cluster died

What it means

Thrown inside RunTask's log-tailing polling loop when `aliveChecks` — one `node::isProcessAlive` per node across all clusters — reports that NOT all node processes are alive. The task tails each node's stdout file and periodically checks liveness; the moment any node exits, it stops streaming and aborts. The error message is intentionally terse because the real cause is in the node's log file that was being streamed.

Source

Thrown at build-tools/src/main/java/org/elasticsearch/gradle/testclusters/RunTask.java:383

                cluster.writeUnicastHostsFiles();
                for (ElasticsearchNode node : cluster.getNodes()) {
                    BufferedReader reader = Files.newBufferedReader(node.getEsOutputFile());
                    toRead.add(reader);
                    aliveChecks.add(node::isProcessAlive);
                }
            }

            while (Thread.currentThread().isInterrupted() == false) {
                boolean readData = false;
                for (BufferedReader bufferedReader : toRead) {
                    if (bufferedReader.ready()) {
                        readData = true;
                        logger.lifecycle(bufferedReader.readLine());
                    }
                }

                if (aliveChecks.stream().allMatch(BooleanSupplier::getAsBoolean) == false) {
                    throw new GradleException("Elasticsearch cluster died");
                }

                if (readData == false) {
                    // no data was ready to be consumed and rather than continuously spinning, pause
                    // for some time to avoid excessive CPU usage. Ideally we would use the JDK
                    // WatchService to receive change notifications but the WatchService does not have
                    // a native MacOS implementation and instead relies upon polling with possible
                    // delays up to 10s before a notification is received. See JDK-7133447.
                    try {
                        Thread.sleep(100L);
                    } catch (InterruptedException e) {
                        Thread.currentThread().interrupt();
                        return;
                    }
                }
            }
        } finally {
            Exception thrown = null;

View on GitHub (pinned to db6a809a66)

Solutions

Read the node log that was being streamed (printed just above the exception) — the actual JVM exit cause is the last lines of that log.
Increase heap or check the host for OOM-killer activity (`dmesg | grep -i kill`) if the log shows `OutOfMemoryError`.
Free the conflicting port or change the configured port; verify no other ES process is running.
Run with `--debug-jvm` to attach a debugger to the node and catch the fatal path, or add `-Dtests.jvm.argline=...` to surface native errors.
If transient, retry; if deterministic, reduce the reproduction to a minimal cluster config and inspect the node's `es_output` file referenced in the reader setup.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  task.runAndWait();
} catch (GradleException e) {
  if (e.getMessage().contains("cluster died")) {
    // surface the node log path, then rethrow or report
    log.error("Node died; inspect the streamed log above and the es_output files.");
  }
  throw e;
}

Prevention

Allocate sufficient heap for the node (`-Dtests.heap.size`) and monitor host memory.
Keep node logs persistent (don't `--no-daemon`-clean them) so the cause is recoverable.
Run dev clusters on hosts with the required native libs and free ports; pre-flight check ports before launch.

When it happens

Trigger: A node JVM exits (OOM, fatal Lucene error, security-manager violation, native lib load failure, port conflict, classpath breakage) while RunTask.runAndWait() is looping. Also triggered if a node is killed externally (signal, OOM-killer) or if startup fails fast before the run loop's first alive check.

Common situations: Running a dev cluster with insufficient heap (`-Dtests.heap.size` too small) so the node OOMs; a plugin on the classpath references a missing class; a port declared in config is already bound; running on a host where required native libs (libicu, zlib) are absent; disk full causing Lucene to abort.

Related errors

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/3a04b8c94153d17d. Report an issue: GitHub.