apache/druid · info · MSQException

CanceledFault(CancellationReason.UNKNOWN)

Error message

CanceledFault(CancellationReason.UNKNOWN)

What it means

WorkerImpl.stop() injects a kernel into the worker's manipulation queue that throws MSQException(CanceledFault.UNKNOWN). This is the worker's shutdown path: any stage still executing is intentionally torn down with UNKNOWN cancellation because the controller asked the worker to stop (e.g. query canceled or shutdown) rather than the query failing on its own.

Solutions

1. If the query result is needed, re-run and let it complete; this fault only reflects an external cancellation, not a data or memory problem
2. Check controller logs / the query's status in the UI to find why the worker was stopped (explicit cancel, controller failure, overlord shutdown)
3. Use the durable shuffle storage / fault tolerance context so cancels and worker loss can be resumed instead of losing the query
4. Ensure clients are not closing connections or issuing sql/sqlStatement cancel calls unintentionally

Example fix

// before: caller treats CanceledFault as an unexpected failure
if (!(e instanceof MSQException) ) throw e;
// after: handle cancellation explicitly
if (e instanceof MSQException && ((MSQException) e).getFault() instanceof CanceledFault) {
  log.info("Query canceled: %s", ((CanceledFault) ((MSQException) e).getFault()).getReason());
  return; // clean up and exit
}

// no pre-check possible; cancellation is external

boolean isCancellation(Throwable e) {
  return e instanceof MSQException && ((MSQException) e).getFault() instanceof CanceledFault;
}

try {
  runQuery(query);
} catch (MSQException e) {
  if (e.getFault() instanceof CanceledFault) {
    log.info("Query canceled: %s", ((CanceledFault) e.getFault()).getReason());
  } else { throw e; }
}

When it happens

Trigger: Controller initiates worker shutdown (query cancel, controller task ending) while the worker's ProcessorAllocator loop is still pulling kernels; the stop kernel executes in KernelManipulationQueue and surfaces CanceledFault(UNKNOWN) to the in-flight stage.

Common situations: User cancels a running SQL query; controller finishes/crashes and kills workers; JVM shutdown of the indexer/peon while MSQ stages are mid-execution; long-running query hitting a controller-side limit.

Related errors

• Broadcast input number out of range
• BroadcastTablesTooLarge
• Can not supply empty segments as input, please use either…
• Cannot accept duplicate stage numbers
• Cannot bucket with partitioning

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/58639e63a0912a2f. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java:707

    final CounterSnapshotsTree retVal = new CounterSnapshotsTree();

    for (final Map.Entry<IntObjectPair<StageId>, CounterTracker> entry : stageCounters.entrySet()) {
      retVal.put(
          entry.getKey().right().getStageNumber(),
          entry.getKey().leftInt(),
          entry.getValue().snapshot()
      );
    }

    return retVal;
  }

  @Override
  public void stop()
  {
    kernelManipulationQueue.add(
        kernel -> {
          throw new MSQException(new CanceledFault(CancellationReason.UNKNOWN));
        }
    );
  }

  /**
   * Returns the context used to create this worker.
   */
  public WorkerContext getWorkerContext()
  {
    return context;
  }

  /**
   * Create a {@link RunWorkOrderListener} for {@link RunWorkOrder} that hooks back into the {@link KernelHolders}
   * in the main loop.
   */
  private RunWorkOrderListener makeRunWorkOrderListener(
      final WorkOrder workOrder,

View on GitHub (pinned to 9b90983fd2)