Could not stop server within %,d millis after unhandled…

Error message

Could not stop server within %,d millis after unhandled Curator error. Halting immediately.

What it means

CuratorModule registers an unhandled-error listener for Curator/Zookeeper: on an unrecoverable ZK session loss it tries to stop the server gracefully, and if the stop does not finish within the grace period it logs this warning and calls Runtime.getRuntime().halt(1), killing the JVM immediately. It exists because a broken ZK session can leave the node in a dangerous half-functional state.

Solutions

Investigate why graceful stop hung: thread dumps before halt, stuck tasks or shutdown hooks
Harden ZooKeeper ensemble health (quorum, session timeout tuned to network latency)
Increase tolerance via curator config (sessionTimeoutMs/connectionTimeoutMs) where appropriate
Treat node restart after halt as expected; enable auto-restart via systemd/supervisor

Defensive patterns

Strategy: fallback

Validate before calling

// preflight: verify ZK reachability before starting Druid nodes
zkCli -server zk1:2181 ls / | head -n 1 || echo 'ZK unreachable'

Try / catch

// cannot be caught — Runtime.halt kills the JVM; guard at ops level
// ensure process supervisor restarts the service automatically
systemd: Restart=always  RestartSec=5

Prevention

Run a healthy ZK quorum (>=3 nodes) with tuned session timeouts
Ensure a supervisor (systemd/k8s) restarts Druid after halt
Capture thread dumps / jstack on hang to diagnose blocked graceful stop
Keep shutdown hooks fast so graceful stop completes within the grace period

When it happens

Trigger: Curator signals ConnectionLossState/unrecoverable error (e.g. ZK session expired permanently), and the graceful stop initiated by the exiter-thread does not complete within ~30 seconds.

Common situations: Long ZK outage causing permanent session expiry; shutdown hooks or in-flight tasks blocking graceful stop; GC pauses or stuck I/O preventing DruidServer stop from finishing; ZK quorum loss in production.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/d4d669f690c15dab. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/curator/CuratorModule.java:173

      final ServiceEmitter emitter,
      final Lifecycle lifecycle
  )
  {
    framework.getUnhandledErrorListenable().addListener((message, e) -> {
      emitter.emit(AlertBuilder.create("Unhandled Curator error").addThrowable(e));
      log.error(e, "Unhandled error in Curator, stopping server.");

      if (haltOnFailedStart) {
        final long startTime = System.currentTimeMillis();
        final Thread halter = new Thread(
            () -> {
              try {
                Threads.sleepFor(30, TimeUnit.SECONDS);
              }
              catch (InterruptedException ignored) {

              }
              log.warn(
                  "Could not stop server within %,d millis after unhandled Curator error. Halting immediately.",
                  System.currentTimeMillis() - startTime
              );
              Runtime.getRuntime().halt(1);
            },
            "exiter-thread"
        );
        halter.setDaemon(true);
        halter.start();
      }

      shutdown(lifecycle);
    });
  }

  /**
   * Add unhandled error listener that shuts down the JVM.
   */

View on GitHub (pinned to 9b90983fd2)