What causes "process was found dead while waiting for {}, {}" in elastic/elasticsearch?

Read the node's log file (the framework logs node output to its es output file) — the JVM exit reason is in the tail.

elastic/elasticsearch · critical · TestClustersException

process was found dead while waiting for {}, {}

Error message

process was found dead while waiting for {}, {}

What it means

Thrown inside TestClusterConfiguration.waitForConditions() when, while polling a wait predicate, `context.isProcessAlive()` returns false before the condition is satisfied. Each iteration first checks process liveness — if the node died mid-wait (e.g. during the `forming cluster` or `http ready` wait), it aborts immediately rather than timing out uselessly. The message names both the condition description and the cluster (`this`).

Source

Thrown at build-tools/src/main/java/org/elasticsearch/gradle/testclusters/TestClusterConfiguration.java:145

    void stop(boolean tailLogs);

    void setNameCustomization(Function<String, String> nameSupplier);

    default void waitForConditions(
        LinkedHashMap<String, Predicate<TestClusterConfiguration>> waitConditions,
        long startedAtMillis,
        long nodeUpTimeout,
        TimeUnit nodeUpTimeoutUnit,
        TestClusterConfiguration context
    ) {
        Logger logger = Logging.getLogger(TestClusterConfiguration.class);
        waitConditions.forEach((description, predicate) -> {
            long thisConditionStartedAt = System.currentTimeMillis();
            boolean conditionMet = false;
            Throwable lastException = null;
            while (System.currentTimeMillis() - startedAtMillis < TimeUnit.MILLISECONDS.convert(nodeUpTimeout, nodeUpTimeoutUnit)) {
                if (context.isProcessAlive() == false) {
                    throw new TestClustersException("process was found dead while waiting for " + description + ", " + this);
                }

                try {
                    if (predicate.test(context)) {
                        conditionMet = true;
                        break;
                    }
                } catch (TestClustersException e) {
                    throw e;
                } catch (Exception e) {
                    lastException = e;
                }
            }
            if (conditionMet == false) {
                String message = String.format(
                    Locale.ROOT,
                    "`%s` failed to wait for %s after %d %s",
                    context,

View on GitHub (pinned to db6a809a66)

Solutions

Read the node's log file (the framework logs node output to its es output file) — the JVM exit reason is in the tail.
Check for bootstrap errors: invalid settings, unresolved plugins, missing security config, port already bound.
If reusing a data directory, clear it (`./gradlew <task> --rerun-tasks` or wipe the testclusters build dir) to rule out corruption.
Increase startup heap if the log shows OOM; verify the seed/unicast hosts files were written (the run task calls `writeUnicastHostsFiles()`).

Defensive patterns

Strategy: try-catch

Try / catch

try {
  cluster.waitForConditions(...);
} catch (TestClustersException e) {
  if (e.getMessage().contains("process was found dead")) {
    // dump the node log tail for post-mortem
    log.error("Node died during startup wait; see node log.");
  }
  throw e;
}

Prevention

Validate cluster config (settings, plugins, security realm) before startup to avoid bootstrap crashes.
Avoid reusing data directories across incompatible versions; clean them between runs.
Ensure required ports are free and seed nodes reachable before the wait begins.

When it happens

Trigger: A node process exits during one of the readiness waits (cluster formation, transport port bound, HTTP endpoint responding, plugin loaded). Any JVM exit — crash, OOM, fatal error, external kill — while a wait condition is still polling triggers this.

Common situations: Node crashes during bootstrap: missing required setting, classpath conflict, security realm misconfiguration, seed-node unreachable, lucene index corruption on a reused data dir, native lib failure. Distinct from #101 (which is the run-task's tailing loop); this is the startup readiness wait.

Related errors

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/b6d0523f0ae4044a. Report an issue: GitHub.