apache/druid · error · IllegalStateException

Worker[%d] not found for stage[%s]

Error message

Worker[%d] not found for stage[%s]

What it means

Thrown in ControllerStageTracker.workOrderSentForWorker when the given worker number has no entry in the tracker's workerToPhase map. The controller records each worker's phase when it starts the stage; reporting a work order sent for an unknown worker means the controller's worker registry and task events are out of sync.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerStageTracker.java:292

      if (ControllerWorkerStagePhase.PRESHUFFLE_WAITING_FOR_RESULT_PARTITION_BOUNDARIES.equals(workerToPhase.get(worker))) {
        workers.add(worker);
      }
    }
    return workers;
  }

  /**
   * Indicates that the work order for worker has been sent. Transitions the state to {@link ControllerWorkerStagePhase#READING_INPUT}
   * if no more work orders need to be sent.
   *
   * @param worker
   */
  void workOrderSentForWorker(int worker)
  {

    workerToPhase.compute(worker, (wk, state) -> {
      if (state == null) {
        throw new ISE("Worker[%d] not found for stage[%s]", wk, stageDef.getStageNumber());
      }
      if (!ControllerWorkerStagePhase.READING_INPUT.canTransitionFrom(state)) {
        throw new ISE(
            "Worker[%d] cannot transistion from state[%s] to state[%s] while sending work order",
            worker,
            state,
            ControllerWorkerStagePhase.READING_INPUT
        );
      }
      return ControllerWorkerStagePhase.READING_INPUT;
    });
    if (phase != ControllerStagePhase.READING_INPUT) {
      if (allWorkOrdersSent()) {
        // if no more work orders need to be sent, change state to reading input from retrying.
        transitionTo(ControllerStagePhase.READING_INPUT);
      }
    }

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Ensure every worker is registered in the stage tracker (via ControllerStagePhase initialization) before calling workOrderSentForWorker.
  2. Guard the call with workerToPhase-style tracking or a containsWorker check.
  3. Collect controller logs and the query id and file a Druid MSQ issue if this occurs in unmodified core code.

Example fix

// before
tracker.workOrderSentForWorker(worker); // throws if unknown
// after
if (tracker.getWorkers().contains(worker)) {
  tracker.workOrderSentForWorker(worker);
}
Defensive patterns

Strategy: validation

Validate before calling

if (tracker.getWorkerCount() > 0 && knownWorkers.contains(worker)) {
  tracker.workOrderSentForWorker(worker);
}

Try / catch

try {
  tracker.workOrderSentForWorker(worker);
} catch (IllegalStateException e) {
  LOG.warn(e, "Ignoring work order ack for unknown worker %d", worker);
}

Prevention

When it happens

Trigger: workOrderSentForWorker(int worker) invoked with a worker number that was never registered via initializeForWorker / stage start, e.g. duplicate or out-of-order work-order tracking, or a stale task report arriving after worker reassignment.

Common situations: MSQ controller crashes/restarts mid-stage with mismatched worker state; duplicate task reports from a retried worker task; custom controller extensions tracking workers incorrectly.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0672a29a4fd1bcbd. Report an issue: GitHub.