{"record":{"id":"58639e63a0912a2f","repo":"apache/druid","slug":"canceledfault-cancellationreason-unknown","errorCode":null,"errorMessage":"CanceledFault(CancellationReason.UNKNOWN)","messagePattern":"CanceledFault\\(CancellationReason\\.UNKNOWN\\)","errorType":"error_code","errorClass":"MSQException","httpStatus":null,"severity":"info","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java","lineNumber":707,"sourceCode":"    final CounterSnapshotsTree retVal = new CounterSnapshotsTree();\n\n    for (final Map.Entry<IntObjectPair<StageId>, CounterTracker> entry : stageCounters.entrySet()) {\n      retVal.put(\n          entry.getKey().right().getStageNumber(),\n          entry.getKey().leftInt(),\n          entry.getValue().snapshot()\n      );\n    }\n\n    return retVal;\n  }\n\n  @Override\n  public void stop()\n  {\n    kernelManipulationQueue.add(\n        kernel -> {\n          throw new MSQException(new CanceledFault(CancellationReason.UNKNOWN));\n        }\n    );\n  }\n\n  /**\n   * Returns the context used to create this worker.\n   */\n  public WorkerContext getWorkerContext()\n  {\n    return context;\n  }\n\n  /**\n   * Create a {@link RunWorkOrderListener} for {@link RunWorkOrder} that hooks back into the {@link KernelHolders}\n   * in the main loop.\n   */\n  private RunWorkOrderListener makeRunWorkOrderListener(\n      final WorkOrder workOrder,","sourceCodeStart":689,"sourceCodeEnd":725,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java#L689-L725","documentation":"WorkerImpl.stop() injects a kernel into the worker's manipulation queue that throws MSQException(CanceledFault.UNKNOWN). This is the worker's shutdown path: any stage still executing is intentionally torn down with UNKNOWN cancellation because the controller asked the worker to stop (e.g. query canceled or shutdown) rather than the query failing on its own.","triggerScenarios":"Controller initiates worker shutdown (query cancel, controller task ending) while the worker's ProcessorAllocator loop is still pulling kernels; the stop kernel executes in KernelManipulationQueue and surfaces CanceledFault(UNKNOWN) to the in-flight stage.","commonSituations":"User cancels a running SQL query; controller finishes/crashes and kills workers; JVM shutdown of the indexer/peon while MSQ stages are mid-execution; long-running query hitting a controller-side limit.","solutions":["If the query result is needed, re-run and let it complete; this fault only reflects an external cancellation, not a data or memory problem","Check controller logs / the query's status in the UI to find why the worker was stopped (explicit cancel, controller failure, overlord shutdown)","Use the durable shuffle storage / fault tolerance context so cancels and worker loss can be resumed instead of losing the query","Ensure clients are not closing connections or issuing sql/sqlStatement cancel calls unintentionally"],"exampleFix":"// before: caller treats CanceledFault as an unexpected failure\nif (!(e instanceof MSQException) ) throw e;\n// after: handle cancellation explicitly\nif (e instanceof MSQException && ((MSQException) e).getFault() instanceof CanceledFault) {\n  log.info(\"Query canceled: %s\", ((CanceledFault) ((MSQException) e).getFault()).getReason());\n  return; // clean up and exit\n}","handlingStrategy":"try-catch","validationCode":"// no pre-check possible; cancellation is external","typeGuard":"boolean isCancellation(Throwable e) {\n  return e instanceof MSQException && ((MSQException) e).getFault() instanceof CanceledFault;\n}","tryCatchPattern":"try {\n  runQuery(query);\n} catch (MSQException e) {\n  if (e.getFault() instanceof CanceledFault) {\n    log.info(\"Query canceled: %s\", ((CanceledFault) e.getFault()).getReason());\n  } else { throw e; }\n}","preventionTips":["Do not cancel queries unintentionally from client timeouts; set realistic query timeouts","Use fault-tolerant MSQ for long-running queries so cancels/worker loss are recoverable","Monitor controller shutdowns and coordinate maintenance windows with query activity"],"tags":["msq","cancellation","query-lifecycle"],"backgroundTag":"operation-cancelled","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}