apache/druid · error · IllegalStateException

LookupCoordinatorManager executor from last start() hasn't…

Error message

LookupCoordinatorManager executor from last start() hasn't finished. Failed to Start.

What it means

Because start() replaces the shared executorService, it first waits up to hostTimeout*10 ms for any executor from a previous start() to terminate. If awaitTermination times out, the old lookup management loop is still running, and start() throws this IllegalStateException to avoid multiple concurrent management executors.

Solutions

1. Wait for the previous stop() to fully complete before calling start() again.
2. Increase lookupCoordinatorManagerConfig hostTimeout so awaitTermination has enough time.
3. Investigate why the previous executor didn't terminate (long-running lookup management task, blocked network calls).
4. Back off leadership start/stop handling to debounce rapid flapping.

Example fix

// before
lookupCoordinatorManager.stop();
lookupCoordinatorManager.start(); // may throw if executor still terminating
// after
lookupCoordinatorManager.stop();
Thread.sleep(lookupCoordinatorManagerConfig.getHostTimeout().getMillis());
lookupCoordinatorManager.start();

// Ensure a previous stop fully completed and enough time elapsed:
await previousStopFuture; // block on stop completion
Thread.sleep(config.getHostTimeout().getMillis());
manager.start();

public static boolean executorTerminated(ExecutorService svc) {
  return svc == null || svc.isTerminated();
}

try {
  manager.start();
} catch (IllegalStateException e) {
  if (e.getMessage().contains("hasn't finished")) {
    Thread.sleep(lookupCoordinatorManagerConfig.getHostTimeout().getMillis() * 10);
    manager.start();
  } else { throw e; }
}

When it happens

Trigger: Calling start() shortly after stop() (or after a failed stop) when the previous scheduled thread pool hasn't shut down within the configured host timeout window.

Common situations: Rapid coordinator leadership flapping (lose/gain in quick succession); tests calling start/stop in tight loops; host timeout configured too small for the management loop to finish its iteration.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• failed to stop, mainThread couldn't finish.
• LookupCoordinatorManager can't start.
• There are too many [ ] pendingNotices.
• Already closed
• Already closed

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a812bdd84fc69de0. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java:396

      if (!lifecycleLock.canStart()) {
        throw new ISE("LookupCoordinatorManager can't start.");
      }

      try {
        LOG.debug("Starting.");

        if (lookupNodeDiscovery == null) {
          lookupNodeDiscovery = new LookupNodeDiscovery(druidNodeDiscoveryProvider);
        }

        //first ensure that previous executorService from last cycle of start/stop has finished completely.
        //so that we don't have multiple live executorService instances lying around doing lookup management.
        if (executorService != null &&
            !executorService.awaitTermination(
                lookupCoordinatorManagerConfig.getHostTimeout().getMillis() * 10,
                TimeUnit.MILLISECONDS
            )) {
          throw new ISE("LookupCoordinatorManager executor from last start() hasn't finished. Failed to Start.");
        }

        executorService = MoreExecutors.listeningDecorator(
            Executors.newScheduledThreadPool(
                lookupCoordinatorManagerConfig.getThreadPoolSize(),
                Execs.makeThreadFactory("LookupCoordinatorManager--%s")
            )
        );

        initializeLookupsConfigWatcher();

        this.backgroundManagerExitedLatch = new CountDownLatch(1);
        this.backgroundManagerFuture = executorService.scheduleWithFixedDelay(
            this::lookupManagementLoop,
            lookupCoordinatorManagerConfig.getInitialDelay(),
            lookupCoordinatorManagerConfig.getPeriod(),
            TimeUnit.MILLISECONDS
        );

View on GitHub (pinned to 9b90983fd2)