apache/druid · error · IllegalStateException

Work order for worker[%d] not found for stage[%d]

Error message

Work order for worker[%d] not found for stage[%d]

What it means

ControllerQueryKernel.getWorkOrder looks up the WorkOrder for a specific (stage, worker) pair in the kernel's stageWorkOrders map. This ISE is thrown when the stage's work-order map exists but contains no entry for the given workerNumber, meaning the controller has no assigned work for that worker at that stage. It is an internal consistency failure: callers should only request work orders for workers that were actually allocated to the stage.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java:717

  {
    ControllerStageTracker stageTracker = stageTrackers.get(stageId);
    if (stageTracker == null) {
      throw new IAE("Cannot find kernel corresponding to stage [%s] in query [%s]", stageId, queryDef.getQueryId());
    }
    return stageTracker;
  }

  private WorkOrder getWorkOrder(int workerNumber, StageId stageId)
  {
    Int2ObjectMap<WorkOrder> stageWorkOrder = stageWorkOrders.get(stageId);

    if (stageWorkOrder == null) {
      throw new ISE("Stage[%d] work orders not found", stageId.getStageNumber());
    }

    WorkOrder workOrder = stageWorkOrder.get(workerNumber);
    if (workOrder == null) {
      throw new ISE("Work order for worker[%d] not found for stage[%d]", workerNumber, stageId.getStageNumber());
    }
    return workOrder;
  }

  /**
   * Whether a given stage is ready to stream results to consumer stages upon transition to "newPhase".
   */
  private boolean readyToReadResults(final StageId stageId, final ControllerStagePhase newPhase)
  {
    if (stageOutputChannelModes.get(stageId) == OutputChannelMode.MEMORY) {
      if (getStageDefinition(stageId).doesSortDuringShuffle()) {
        // Sorting stages start producing output when they finish reading their input.
        return newPhase.isDoneReadingInput();
      } else {
        // Non-sorting stages start producing output immediately.
        return newPhase == ControllerStagePhase.NEW;
      }
    } else {

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify the workerNumber is within the stage's allocated worker count (0..StageDefinition.getMaxWorkerCount()-1) before lookup.
2. Check whether fault tolerance/worker retry reallocated or removed the worker; use the kernel's worker-selection API instead of assuming the original worker number.
3. Confirm the stage id belongs to the same query definition/epoch as the kernel's stageWorkOrders map (no stale StageId after controller restart).
4. If reproducible without failures, capture the controller log at level DEBUG and file a Druid MSQ bug with the query id — this indicates a kernel bookkeeping bug.

Example fix

// before
WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);

// after
if (workerNumber < stageDef.getMaxWorkerCount()) {
  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);
} else {
  workerNumber = pickEligibleWorker(stageId);
  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);
}

if (workerNumber < 0 || workerNumber >= stageDef.getMaxWorkerCount()) {
  throw new IllegalArgumentException("workerNumber out of range for stage " + stageId);
}

try {
  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);
} catch (IllegalStateException e) {
  // reassign an eligible worker or treat as kernel inconsistency
  workerNumber = kernel.selectWorkerForStage(stageId);
}

When it happens

Trigger: Calling getWorkOrder(workerNumber, stageId) where stageWorkOrders.get(stageId) is non-null but stageWorkOrders.get(stageId).get(workerNumber) returns null — e.g. a worker number that was never allocated for that stage, a worker removed during retry/fault tolerance and its entry dropped, or a stage id/worker numbering mismatch (kernel recreated with different worker count).

Common situations: MSQ controller logs during worker retry handling (getWorkInCaseWorkerEligibleForRetry path), after partial worker failures cause reallocation, or when a controller restart restores a kernel snapshot with fewer work orders than workers referenced by runtime code.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• Result partition information is not ready yet
• Number of partitions not known for [%s] with maxPartitions[%
• Cannot accept duplicate stage numbers
• Work order not present for stage[%s]
• Cannot start the stage: [%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/961c486d89c6c0ce. Report an issue: GitHub.