{"record":{"id":"dbc68b63a7c8bc8c","repo":"apache/druid","slug":"stage-d-work-orders-not-found","errorCode":null,"errorMessage":"Stage[%d] work orders not found","messagePattern":"Stage\\[(.+?)\\] work orders not found","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java","lineNumber":712,"sourceCode":"\n  /**\n   * Fetches and returns the stage kernel corresponding to the provided stage id, else throws {@link IAE}\n   */\n  private ControllerStageTracker getStageTrackerOrThrow(StageId stageId)\n  {\n    ControllerStageTracker stageTracker = stageTrackers.get(stageId);\n    if (stageTracker == null) {\n      throw new IAE(\"Cannot find kernel corresponding to stage [%s] in query [%s]\", stageId, queryDef.getQueryId());\n    }\n    return stageTracker;\n  }\n\n  private WorkOrder getWorkOrder(int workerNumber, StageId stageId)\n  {\n    Int2ObjectMap<WorkOrder> stageWorkOrder = stageWorkOrders.get(stageId);\n\n    if (stageWorkOrder == null) {\n      throw new ISE(\"Stage[%d] work orders not found\", stageId.getStageNumber());\n    }\n\n    WorkOrder workOrder = stageWorkOrder.get(workerNumber);\n    if (workOrder == null) {\n      throw new ISE(\"Work order for worker[%d] not found for stage[%d]\", workerNumber, stageId.getStageNumber());\n    }\n    return workOrder;\n  }\n\n  /**\n   * Whether a given stage is ready to stream results to consumer stages upon transition to \"newPhase\".\n   */\n  private boolean readyToReadResults(final StageId stageId, final ControllerStagePhase newPhase)\n  {\n    if (stageOutputChannelModes.get(stageId) == OutputChannelMode.MEMORY) {\n      if (getStageDefinition(stageId).doesSortDuringShuffle()) {\n        // Sorting stages start producing output when they finish reading their input.\n        return newPhase.isDoneReadingInput();","sourceCodeStart":694,"sourceCodeEnd":730,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java#L694-L730","documentation":"Thrown by ControllerQueryKernel.getWorkOrder as an IllegalStateException when the stageWorkOrders map has no Int2ObjectMap for the stage at all, so no per-worker work order can be fetched. Work orders are removed by finishStage, so this means the stage is no longer in an active, work-order-bearing state.","triggerScenarios":"Calling getWorkOrder via getWorkInCaseWorkerEligibleForRetry for a stage whose work orders were never created with createWorkOrders, or after finishStage removed them (stage already finished), or for a failed stage whose orders were cleared.","commonSituations":"Worker retry/encounter logic asking for the work order of a stage that has just finished or failed; race between stage completion (finishStage clearing stageWorkOrders) and a late retry request from a worker.","solutions":["Check that the stage is active (not finished/failed) before requesting a retry work order","Synchronize retry handling with the state machine so getWorkOrder is only called while the stage is in a work-bearing phase","If the work order is legitimately gone, fail or requeue the retry rather than fetching a stale order","Verify createWorkOrders ran for this stage number before any worker retries are served"],"exampleFix":"// before\nWorkOrder order = kernel.getWorkOrder(workerNumber, stageId);\n\n// after\nif (kernel.getStagePhase(stageId).isWorkOrderPresent()) {\n  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);\n} else {\n  // stage already finished; do not retry with a stale work order\n}","handlingStrategy":"validation","validationCode":"if (queryKernel.getStagePhase(stageId) != ControllerStagePhase.FINISHED\n    && queryKernel.isStageKnown(stageId)) {\n  WorkOrder order = queryKernel.getWorkOrder(workerNumber, stageId);\n}","typeGuard":null,"tryCatchPattern":"try {\n  return queryKernel.getWorkOrder(workerNumber, stageId);\n} catch (IllegalStateException e) {\n  LOG.warn(e, \"Work orders gone for stage %s; dropping retry\", stageId);\n  return null;\n}","preventionTips":["Serve worker retries only while the stage is in an active work-bearing phase","Remember finishStage clears stageWorkOrders; do not fetch orders afterwards","Confirm createWorkOrders ran before accepting retry requests","Serialize retry handling with the state machine to avoid completion/retry races"],"tags":["msq","work-order","retry","controller"],"backgroundTag":"record-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}