apache/druid · error · IllegalStateException

Controller kernel queue is full. Main controller loop may…

Error message

Controller kernel queue is full. Main controller loop may be delayed or stuck.

What it means

An MSQ controller tries to enqueue a kernel manipulation (Consumer of ControllerQueryKernel) onto the controller's single-threaded kernel manipulation queue, but the bounded queue is full. This means the main controller loop is not draining the queue fast enough or is stuck, so the method throws IllegalStateException.

Solutions

1. Inspect controller logs around the error for the reason the main loop stalled (slow stage completion, export, task failures)
2. Check for controller task failures/crashes in worker logs and Overlord task list; re-run the query
3. Avoid extremely frequent counter updates; look for upstream errors delaying stage completion
4. If reproducible, file an issue with controller logs — a full queue indicates an internal liveness bug

try { controller.addToKernelManipulationQueue(kernelConsumer); } catch (IllegalStateException e) { if (e.getMessage().contains("kernel queue is full")) { /* treat query as failed; restart/re-run query after checking controller health */ } }

When it happens

Trigger: Calling addToKernelManipulationQueue (directly or via stop, workerError, doneReadingInput, updateCounters, updatePartialKeyStatisticsInformation, getWorkerFailureListener) when the ArrayBlockingQueue offer fails because the controller's main loop is blocked or dead.

Common situations: Main controller loop blocked on slow I/O (e.g. writing results, export, counter persistence); controller thread crashed without shutting down workers; deadlock or severe GC pauses; very high-frequency counter updates flooding the queue.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Attempt to add row to swapped-out sink for segment
• Background lookup manager thread could not be cancelled
• Broadcast input number out of range
• BroadcastTablesTooLarge
• Can not supply empty segments as input, please use either…

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/d98fd70464f4de0b. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java:717

      throw new IOException("Failed to release locks", e);
    }
  }

  /**
   * Adds some logic to {@link #kernelManipulationQueue}, where it will, in due time, be executed by the main
   * controller loop in {@link RunQueryUntilDone#run()}.
   * <p>
   * If the consumer throws an exception, the query fails.
   * <p>
   * Consumers must not perform blocking operations (network calls, waiting on futures, sleeping, etc.), because
   * the main controller loop executes them in sequence and blocking would delay controller operations.
   */
  public void addToKernelManipulationQueue(Consumer<ControllerQueryKernel> kernelConsumer)
  {
    if (!kernelManipulationQueue.offer(kernelConsumer)) {
      final String message = "Controller kernel queue is full. Main controller loop may be delayed or stuck.";
      log.warn(message);
      throw new IllegalStateException(message);
    }
  }

  public static void ensureExportLocationEmpty(final ControllerContext context, final MSQDestination destination)
  {
    if (MSQControllerTask.isExport(destination)) {
      final ExportMSQDestination exportMSQDestination = (ExportMSQDestination) destination;
      final ExportStorageProvider exportStorageProvider = exportMSQDestination.getExportStorageProvider();

      try {
        // Check that the export destination is empty as a sanity check. We want
        // to avoid modifying any other files with export.
        Iterator<String> filesIterator = exportStorageProvider.createStorageConnector(context.taskTempDir())
            .listDir("");
        if (filesIterator.hasNext()) {
          throw DruidException.forPersona(DruidException.Persona.USER)
              .ofCategory(DruidException.Category.RUNTIME_FAILURE)
              .build(

View on GitHub (pinned to 9b90983fd2)