{"record":{"id":"d4d669f690c15dab","repo":"apache/druid","slug":"could-not-stop-server-within-d-millis-after-unha","errorCode":null,"errorMessage":"Could not stop server within %,d millis after unhandled Curator error. Halting immediately.","messagePattern":"Could not stop server within %,d millis after unhandled Curator error\\. Halting immediately\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"server/src/main/java/org/apache/druid/curator/CuratorModule.java","lineNumber":173,"sourceCode":"      final ServiceEmitter emitter,\n      final Lifecycle lifecycle\n  )\n  {\n    framework.getUnhandledErrorListenable().addListener((message, e) -> {\n      emitter.emit(AlertBuilder.create(\"Unhandled Curator error\").addThrowable(e));\n      log.error(e, \"Unhandled error in Curator, stopping server.\");\n\n      if (haltOnFailedStart) {\n        final long startTime = System.currentTimeMillis();\n        final Thread halter = new Thread(\n            () -> {\n              try {\n                Threads.sleepFor(30, TimeUnit.SECONDS);\n              }\n              catch (InterruptedException ignored) {\n\n              }\n              log.warn(\n                  \"Could not stop server within %,d millis after unhandled Curator error. Halting immediately.\",\n                  System.currentTimeMillis() - startTime\n              );\n              Runtime.getRuntime().halt(1);\n            },\n            \"exiter-thread\"\n        );\n        halter.setDaemon(true);\n        halter.start();\n      }\n\n      shutdown(lifecycle);\n    });\n  }\n\n  /**\n   * Add unhandled error listener that shuts down the JVM.\n   */","sourceCodeStart":155,"sourceCodeEnd":191,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/curator/CuratorModule.java#L155-L191","documentation":"CuratorModule registers an unhandled-error listener for Curator/Zookeeper: on an unrecoverable ZK session loss it tries to stop the server gracefully, and if the stop does not finish within the grace period it logs this warning and calls Runtime.getRuntime().halt(1), killing the JVM immediately. It exists because a broken ZK session can leave the node in a dangerous half-functional state.","triggerScenarios":"Curator signals ConnectionLossState/unrecoverable error (e.g. ZK session expired permanently), and the graceful stop initiated by the exiter-thread does not complete within ~30 seconds.","commonSituations":"Long ZK outage causing permanent session expiry; shutdown hooks or in-flight tasks blocking graceful stop; GC pauses or stuck I/O preventing DruidServer stop from finishing; ZK quorum loss in production.","solutions":["Investigate why graceful stop hung: thread dumps before halt, stuck tasks or shutdown hooks","Harden ZooKeeper ensemble health (quorum, session timeout tuned to network latency)","Increase tolerance via curator config (sessionTimeoutMs/connectionTimeoutMs) where appropriate","Treat node restart after halt as expected; enable auto-restart via systemd/supervisor"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// preflight: verify ZK reachability before starting Druid nodes\nzkCli -server zk1:2181 ls / | head -n 1 || echo 'ZK unreachable'","typeGuard":null,"tryCatchPattern":"// cannot be caught — Runtime.halt kills the JVM; guard at ops level\n// ensure process supervisor restarts the service automatically\nsystemd: Restart=always  RestartSec=5","preventionTips":["Run a healthy ZK quorum (>=3 nodes) with tuned session timeouts","Ensure a supervisor (systemd/k8s) restarts Druid after halt","Capture thread dumps / jstack on hang to diagnose blocked graceful stop","Keep shutdown hooks fast so graceful stop completes within the grace period"],"tags":["zookeeper","curator","shutdown","jvm-halt"],"backgroundTag":"connection-refused","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}