apache/druid · error · IllegalStateException

Stage[%d] work orders not found

Error message

Stage[%d] work orders not found

What it means

Thrown by ControllerQueryKernel.getWorkOrder as an IllegalStateException when the stageWorkOrders map has no Int2ObjectMap for the stage at all, so no per-worker work order can be fetched. Work orders are removed by finishStage, so this means the stage is no longer in an active, work-order-bearing state.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java:712

  /**
   * Fetches and returns the stage kernel corresponding to the provided stage id, else throws {@link IAE}
   */
  private ControllerStageTracker getStageTrackerOrThrow(StageId stageId)
  {
    ControllerStageTracker stageTracker = stageTrackers.get(stageId);
    if (stageTracker == null) {
      throw new IAE("Cannot find kernel corresponding to stage [%s] in query [%s]", stageId, queryDef.getQueryId());
    }
    return stageTracker;
  }

  private WorkOrder getWorkOrder(int workerNumber, StageId stageId)
  {
    Int2ObjectMap<WorkOrder> stageWorkOrder = stageWorkOrders.get(stageId);

    if (stageWorkOrder == null) {
      throw new ISE("Stage[%d] work orders not found", stageId.getStageNumber());
    }

    WorkOrder workOrder = stageWorkOrder.get(workerNumber);
    if (workOrder == null) {
      throw new ISE("Work order for worker[%d] not found for stage[%d]", workerNumber, stageId.getStageNumber());
    }
    return workOrder;
  }

  /**
   * Whether a given stage is ready to stream results to consumer stages upon transition to "newPhase".
   */
  private boolean readyToReadResults(final StageId stageId, final ControllerStagePhase newPhase)
  {
    if (stageOutputChannelModes.get(stageId) == OutputChannelMode.MEMORY) {
      if (getStageDefinition(stageId).doesSortDuringShuffle()) {
        // Sorting stages start producing output when they finish reading their input.
        return newPhase.isDoneReadingInput();

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Check that the stage is active (not finished/failed) before requesting a retry work order
2. Synchronize retry handling with the state machine so getWorkOrder is only called while the stage is in a work-bearing phase
3. If the work order is legitimately gone, fail or requeue the retry rather than fetching a stale order
4. Verify createWorkOrders ran for this stage number before any worker retries are served

Example fix

// before
WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);

// after
if (kernel.getStagePhase(stageId).isWorkOrderPresent()) {
  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);
} else {
  // stage already finished; do not retry with a stale work order
}

if (queryKernel.getStagePhase(stageId) != ControllerStagePhase.FINISHED
    && queryKernel.isStageKnown(stageId)) {
  WorkOrder order = queryKernel.getWorkOrder(workerNumber, stageId);
}

try {
  return queryKernel.getWorkOrder(workerNumber, stageId);
} catch (IllegalStateException e) {
  LOG.warn(e, "Work orders gone for stage %s; dropping retry", stageId);
  return null;
}

When it happens

Trigger: Calling getWorkOrder via getWorkInCaseWorkerEligibleForRetry for a stage whose work orders were never created with createWorkOrders, or after finishStage removed them (stage already finished), or for a failed stage whose orders were cleared.

Common situations: Worker retry/encounter logic asking for the work order of a stage that has just finished or failed; race between stage completion (finishStage clearing stageWorkOrders) and a late retry request from a worker.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• Work order not present for stage[%s]
• Cannot start the stage: [%s]
• Cannot mark the stage: [%s] finished
• Work order for worker[%d] not found for stage[%d]
• Result partition information is not ready yet

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/dbc68b63a7c8bc8c. Report an issue: GitHub.