{"record":{"id":"3a04b8c94153d17d","repo":"elastic/elasticsearch","slug":"elasticsearch-cluster-died","errorCode":null,"errorMessage":"Elasticsearch cluster died","messagePattern":"Elasticsearch cluster died","errorType":"exception","errorClass":"GradleException","httpStatus":null,"severity":"critical","filePath":"build-tools/src/main/java/org/elasticsearch/gradle/testclusters/RunTask.java","lineNumber":383,"sourceCode":"                cluster.writeUnicastHostsFiles();\n                for (ElasticsearchNode node : cluster.getNodes()) {\n                    BufferedReader reader = Files.newBufferedReader(node.getEsOutputFile());\n                    toRead.add(reader);\n                    aliveChecks.add(node::isProcessAlive);\n                }\n            }\n\n            while (Thread.currentThread().isInterrupted() == false) {\n                boolean readData = false;\n                for (BufferedReader bufferedReader : toRead) {\n                    if (bufferedReader.ready()) {\n                        readData = true;\n                        logger.lifecycle(bufferedReader.readLine());\n                    }\n                }\n\n                if (aliveChecks.stream().allMatch(BooleanSupplier::getAsBoolean) == false) {\n                    throw new GradleException(\"Elasticsearch cluster died\");\n                }\n\n                if (readData == false) {\n                    // no data was ready to be consumed and rather than continuously spinning, pause\n                    // for some time to avoid excessive CPU usage. Ideally we would use the JDK\n                    // WatchService to receive change notifications but the WatchService does not have\n                    // a native MacOS implementation and instead relies upon polling with possible\n                    // delays up to 10s before a notification is received. See JDK-7133447.\n                    try {\n                        Thread.sleep(100L);\n                    } catch (InterruptedException e) {\n                        Thread.currentThread().interrupt();\n                        return;\n                    }\n                }\n            }\n        } finally {\n            Exception thrown = null;","sourceCodeStart":365,"sourceCodeEnd":401,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/RunTask.java#L365-L401","documentation":"Thrown inside RunTask's log-tailing polling loop when `aliveChecks` — one `node::isProcessAlive` per node across all clusters — reports that NOT all node processes are alive. The task tails each node's stdout file and periodically checks liveness; the moment any node exits, it stops streaming and aborts. The error message is intentionally terse because the real cause is in the node's log file that was being streamed.","triggerScenarios":"A node JVM exits (OOM, fatal Lucene error, security-manager violation, native lib load failure, port conflict, classpath breakage) while RunTask.runAndWait() is looping. Also triggered if a node is killed externally (signal, OOM-killer) or if startup fails fast before the run loop's first alive check.","commonSituations":"Running a dev cluster with insufficient heap (`-Dtests.heap.size` too small) so the node OOMs; a plugin on the classpath references a missing class; a port declared in config is already bound; running on a host where required native libs (libicu, zlib) are absent; disk full causing Lucene to abort.","solutions":["Read the node log that was being streamed (printed just above the exception) — the actual JVM exit cause is the last lines of that log.","Increase heap or check the host for OOM-killer activity (`dmesg | grep -i kill`) if the log shows `OutOfMemoryError`.","Free the conflicting port or change the configured port; verify no other ES process is running.","Run with `--debug-jvm` to attach a debugger to the node and catch the fatal path, or add `-Dtests.jvm.argline=...` to surface native errors.","If transient, retry; if deterministic, reduce the reproduction to a minimal cluster config and inspect the node's `es_output` file referenced in the reader setup."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  task.runAndWait();\n} catch (GradleException e) {\n  if (e.getMessage().contains(\"cluster died\")) {\n    // surface the node log path, then rethrow or report\n    log.error(\"Node died; inspect the streamed log above and the es_output files.\");\n  }\n  throw e;\n}","preventionTips":["Allocate sufficient heap for the node (`-Dtests.heap.size`) and monitor host memory.","Keep node logs persistent (don't `--no-daemon`-clean them) so the cause is recoverable.","Run dev clusters on hosts with the required native libs and free ports; pre-flight check ports before launch."],"tags":["gradle","testclusters","process-lifecycle","runtime","build-tools"],"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T12:17:08.281Z"}