apache/druid · error · IllegalStateException

Worker[ ] not found for stage[ ]

Error message

Worker[%d] not found for stage[%s]

What it means

Thrown in ControllerStageTracker.workOrderSentForWorker when the given worker number has no entry in the tracker's workerToPhase map. The controller records each worker's phase when it starts the stage; reporting a work order sent for an unknown worker means the controller's worker registry and task events are out of sync.

Solutions

1. Ensure every worker is registered in the stage tracker (via ControllerStagePhase initialization) before calling workOrderSentForWorker.
2. Guard the call with workerToPhase-style tracking or a containsWorker check.
3. Collect controller logs and the query id and file a Druid MSQ issue if this occurs in unmodified core code.

Example fix

// before
tracker.workOrderSentForWorker(worker); // throws if unknown
// after
if (tracker.getWorkers().contains(worker)) {
  tracker.workOrderSentForWorker(worker);
}

if (tracker.getWorkerCount() > 0 && knownWorkers.contains(worker)) {
  tracker.workOrderSentForWorker(worker);
}

try {
  tracker.workOrderSentForWorker(worker);
} catch (IllegalStateException e) {
  LOG.warn(e, "Ignoring work order ack for unknown worker %d", worker);
}

When it happens

Trigger: workOrderSentForWorker(int worker) invoked with a worker number that was never registered via initializeForWorker / stage start, e.g. duplicate or out-of-order work-order tracking, or a stale task report arriving after worker reassignment.

Common situations: MSQ controller crashes/restarts mid-stage with mismatched worker state; duplicate task reports from a retried worker task; custom controller extensions tracking workers incorrectly.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• Controller
• Worker
• BroadcastTablesTooLarge
• Cannot handle dataSource
• Cannot mark the stage

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0672a29a4fd1bcbd. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerStageTracker.java:292

      if (ControllerWorkerStagePhase.PRESHUFFLE_WAITING_FOR_RESULT_PARTITION_BOUNDARIES.equals(workerToPhase.get(worker))) {
        workers.add(worker);
      }
    }
    return workers;
  }

  /**
   * Indicates that the work order for worker has been sent. Transitions the state to {@link ControllerWorkerStagePhase#READING_INPUT}
   * if no more work orders need to be sent.
   *
   * @param worker
   */
  void workOrderSentForWorker(int worker)
  {

    workerToPhase.compute(worker, (wk, state) -> {
      if (state == null) {
        throw new ISE("Worker[%d] not found for stage[%s]", wk, stageDef.getStageNumber());
      }
      if (!ControllerWorkerStagePhase.READING_INPUT.canTransitionFrom(state)) {
        throw new ISE(
            "Worker[%d] cannot transistion from state[%s] to state[%s] while sending work order",
            worker,
            state,
            ControllerWorkerStagePhase.READING_INPUT
        );
      }
      return ControllerWorkerStagePhase.READING_INPUT;
    });
    if (phase != ControllerStagePhase.READING_INPUT) {
      if (allWorkOrdersSent()) {
        // if no more work orders need to be sent, change state to reading input from retrying.
        transitionTo(ControllerStagePhase.READING_INPUT);
      }
    }

View on GitHub (pinned to 9b90983fd2)