elastic/elasticsearch · error · TestClustersException

Was not able to terminate elasticsearch process for {}

Error message

Was not able to terminate elasticsearch process for {}

What it means

Thrown when an Elasticsearch testcluster process is still alive after BOTH a graceful destroy() and a destroyForcibly() followed by a waitForProcessToExit (bounded by ES_DESTROY_TIMEOUT = 20 seconds). The plugin has exhausted its shutdown escalation path and the PID will leak. The thrown exception has no cause — the failure is a stuck process, not an IOException.

Source

Thrown at build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java:1102

        List<ProcessHandle> children = processHandle.children().toList();
        try {
            logProcessInfo("Terminating elasticsearch process" + (forcibly ? " forcibly " : "gracefully") + ":", processHandle.info());

            if (forcibly) {
                processHandle.destroyForcibly();
            } else {
                processHandle.destroy();
                waitForProcessToExit(processHandle);
                if (processHandle.isAlive() == false) {
                    return;
                }
                LOGGER.info("process did not terminate after {} {}, stopping it forcefully", ES_DESTROY_TIMEOUT, ES_DESTROY_TIMEOUT_UNIT);
                processHandle.destroyForcibly();
            }

            waitForProcessToExit(processHandle);
            if (processHandle.isAlive()) {
                throw new TestClustersException("Was not able to terminate elasticsearch process for " + this);
            }
        } finally {
            children.forEach(each -> stopHandle(each, forcibly));
        }
    }

    private void logProcessInfo(String prefix, ProcessHandle.Info info) {
        LOGGER.info(
            prefix + " commandLine:`{}` command:`{}` args:`{}`",
            info.commandLine().orElse("-"),
            info.command().orElse("-"),
            Arrays.stream(info.arguments().orElse(new String[] {})).map(each -> "'" + each + "'").collect(Collectors.joining(" "))
        );
    }

    private void logFileContents(String description, Path from, boolean tailLogs) {
        final Map<String, Pair<String, Integer>> errorsAndWarnings = new LinkedHashMap<>();
        LinkedList<String> ring = new LinkedList<>();

View on GitHub (pinned to db6a809a66)

Solutions

1. Capture the processHandle.info() log line (printed just before the throw) to identify the stuck PID and its command line.
2. Inspect the node's log file (esOutputFile) for the last actions before the hang — this is the real root cause.
3. Kill the leaked PID manually: kill -9 <pid> (Unix) or taskkill /F /PID <pid> (Windows), then re-run the test.
4. If reproducible, run the test with --debug-jvm and attach a debugger/profiler to the ES JVM to find the deadlock.
5. Increase ES_DESTROY_TIMEOUT only if you have evidence the process does exit given more time; otherwise you are masking a real hang.

Example fix

// after a leaked process, manually reap it:
// jps -lv   # find the Elasticsearch pid
// kill -9 <pid>
// then: ./gradlew :server:internalClusterTest --tests YourTest

When it happens

Trigger: stopHandle() calls processHandle.destroy(); if still alive after the wait, it calls destroyForcibly(), waits again, then throws if processHandle.isAlive() is still true. Common when the ES JVM is wedged in a native call, blocked on a socket, or a wrapper process (Windows) ignored SIGKILL-equivalent.

Common situations: Long-running integration tests where ES is stuck in a GC death spiral, an ML or watcher thread holding a lock, antivirus scanning the JVM on Windows preventing termination, or a kernel-level unkillable process (D-state on I/O). Often a symptom of a real ES hang, not a plugin bug.

Related errors

• Failed to start ES process for {}
• Configuration for {} can not be altered, already locked
• Task {} is not configured to use any clusters. Be sure to ca
• Elasticsearch cluster died
• Can not use {} with {}

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/d437bfdfd11fd344. Report an issue: GitHub.