{"record":{"id":"a812bdd84fc69de0","repo":"apache/druid","slug":"lookupcoordinatormanager-executor-from-last-start","errorCode":null,"errorMessage":"LookupCoordinatorManager executor from last start() hasn't finished. Failed to Start.","messagePattern":"LookupCoordinatorManager executor from last start\\(\\) hasn't finished\\. Failed to Start\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java","lineNumber":396,"sourceCode":"      if (!lifecycleLock.canStart()) {\n        throw new ISE(\"LookupCoordinatorManager can't start.\");\n      }\n\n      try {\n        LOG.debug(\"Starting.\");\n\n        if (lookupNodeDiscovery == null) {\n          lookupNodeDiscovery = new LookupNodeDiscovery(druidNodeDiscoveryProvider);\n        }\n\n        //first ensure that previous executorService from last cycle of start/stop has finished completely.\n        //so that we don't have multiple live executorService instances lying around doing lookup management.\n        if (executorService != null &&\n            !executorService.awaitTermination(\n                lookupCoordinatorManagerConfig.getHostTimeout().getMillis() * 10,\n                TimeUnit.MILLISECONDS\n            )) {\n          throw new ISE(\"LookupCoordinatorManager executor from last start() hasn't finished. Failed to Start.\");\n        }\n\n        executorService = MoreExecutors.listeningDecorator(\n            Executors.newScheduledThreadPool(\n                lookupCoordinatorManagerConfig.getThreadPoolSize(),\n                Execs.makeThreadFactory(\"LookupCoordinatorManager--%s\")\n            )\n        );\n\n        initializeLookupsConfigWatcher();\n\n        this.backgroundManagerExitedLatch = new CountDownLatch(1);\n        this.backgroundManagerFuture = executorService.scheduleWithFixedDelay(\n            this::lookupManagementLoop,\n            lookupCoordinatorManagerConfig.getInitialDelay(),\n            lookupCoordinatorManagerConfig.getPeriod(),\n            TimeUnit.MILLISECONDS\n        );","sourceCodeStart":378,"sourceCodeEnd":414,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java#L378-L414","documentation":"Because start() replaces the shared executorService, it first waits up to hostTimeout*10 ms for any executor from a previous start() to terminate. If awaitTermination times out, the old lookup management loop is still running, and start() throws this IllegalStateException to avoid multiple concurrent management executors.","triggerScenarios":"Calling start() shortly after stop() (or after a failed stop) when the previous scheduled thread pool hasn't shut down within the configured host timeout window.","commonSituations":"Rapid coordinator leadership flapping (lose/gain in quick succession); tests calling start/stop in tight loops; host timeout configured too small for the management loop to finish its iteration.","solutions":["Wait for the previous stop() to fully complete before calling start() again.","Increase lookupCoordinatorManagerConfig hostTimeout so awaitTermination has enough time.","Investigate why the previous executor didn't terminate (long-running lookup management task, blocked network calls).","Back off leadership start/stop handling to debounce rapid flapping."],"exampleFix":"// before\nlookupCoordinatorManager.stop();\nlookupCoordinatorManager.start(); // may throw if executor still terminating\n// after\nlookupCoordinatorManager.stop();\nThread.sleep(lookupCoordinatorManagerConfig.getHostTimeout().getMillis());\nlookupCoordinatorManager.start();","handlingStrategy":"retry","validationCode":"// Ensure a previous stop fully completed and enough time elapsed:\nawait previousStopFuture; // block on stop completion\nThread.sleep(config.getHostTimeout().getMillis());\nmanager.start();","typeGuard":"public static boolean executorTerminated(ExecutorService svc) {\n  return svc == null || svc.isTerminated();\n}","tryCatchPattern":"try {\n  manager.start();\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"hasn't finished\")) {\n    Thread.sleep(lookupCoordinatorManagerConfig.getHostTimeout().getMillis() * 10);\n    manager.start();\n  } else { throw e; }\n}","preventionTips":["Allow the full host-timeout window after stop() before calling start() again.","Debounce rapid leadership flapping so start/stop pairs are not issued back-to-back.","Increase hostTimeout in the lookup coordinator config if the management loop needs longer to drain.","Ensure no long-running task keeps the previous executor from terminating."],"tags":["lifecycle","executor","threading","lookups"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}