{"record":{"id":"d98fd70464f4de0b","repo":"apache/druid","slug":"controller-kernel-queue-is-full-main-controller-l","errorCode":null,"errorMessage":"Controller kernel queue is full. Main controller loop may be delayed or stuck.","messagePattern":"Controller kernel queue is full\\. Main controller loop may be delayed or stuck\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java","lineNumber":717,"sourceCode":"      throw new IOException(\"Failed to release locks\", e);\n    }\n  }\n\n  /**\n   * Adds some logic to {@link #kernelManipulationQueue}, where it will, in due time, be executed by the main\n   * controller loop in {@link RunQueryUntilDone#run()}.\n   * <p>\n   * If the consumer throws an exception, the query fails.\n   * <p>\n   * Consumers must not perform blocking operations (network calls, waiting on futures, sleeping, etc.), because\n   * the main controller loop executes them in sequence and blocking would delay controller operations.\n   */\n  public void addToKernelManipulationQueue(Consumer<ControllerQueryKernel> kernelConsumer)\n  {\n    if (!kernelManipulationQueue.offer(kernelConsumer)) {\n      final String message = \"Controller kernel queue is full. Main controller loop may be delayed or stuck.\";\n      log.warn(message);\n      throw new IllegalStateException(message);\n    }\n  }\n\n  public static void ensureExportLocationEmpty(final ControllerContext context, final MSQDestination destination)\n  {\n    if (MSQControllerTask.isExport(destination)) {\n      final ExportMSQDestination exportMSQDestination = (ExportMSQDestination) destination;\n      final ExportStorageProvider exportStorageProvider = exportMSQDestination.getExportStorageProvider();\n\n      try {\n        // Check that the export destination is empty as a sanity check. We want\n        // to avoid modifying any other files with export.\n        Iterator<String> filesIterator = exportStorageProvider.createStorageConnector(context.taskTempDir())\n            .listDir(\"\");\n        if (filesIterator.hasNext()) {\n          throw DruidException.forPersona(DruidException.Persona.USER)\n              .ofCategory(DruidException.Category.RUNTIME_FAILURE)\n              .build(","sourceCodeStart":699,"sourceCodeEnd":735,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java#L699-L735","documentation":"An MSQ controller tries to enqueue a kernel manipulation (Consumer of ControllerQueryKernel) onto the controller's single-threaded kernel manipulation queue, but the bounded queue is full. This means the main controller loop is not draining the queue fast enough or is stuck, so the method throws IllegalStateException.","triggerScenarios":"Calling addToKernelManipulationQueue (directly or via stop, workerError, doneReadingInput, updateCounters, updatePartialKeyStatisticsInformation, getWorkerFailureListener) when the ArrayBlockingQueue offer fails because the controller's main loop is blocked or dead.","commonSituations":"Main controller loop blocked on slow I/O (e.g. writing results, export, counter persistence); controller thread crashed without shutting down workers; deadlock or severe GC pauses; very high-frequency counter updates flooding the queue.","solutions":["Inspect controller logs around the error for the reason the main loop stalled (slow stage completion, export, task failures)","Check for controller task failures/crashes in worker logs and Overlord task list; re-run the query","Avoid extremely frequent counter updates; look for upstream errors delaying stage completion","If reproducible, file an issue with controller logs — a full queue indicates an internal liveness bug"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { controller.addToKernelManipulationQueue(kernelConsumer); } catch (IllegalStateException e) { if (e.getMessage().contains(\"kernel queue is full\")) { /* treat query as failed; restart/re-run query after checking controller health */ } }","preventionTips":["Avoid flooding controllers with very high-frequency counter updates","Monitor controller task liveness and restart stuck queries","Report reproducible queue-full states as controller liveness bugs"],"tags":["msq","concurrency","queue-full"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}