{"record":{"id":"961c486d89c6c0ce","repo":"apache/druid","slug":"work-order-for-worker-d-not-found-for-stage-d","errorCode":null,"errorMessage":"Work order for worker[%d] not found for stage[%d]","messagePattern":"Work order for worker\\[(.+?)\\] not found for stage\\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java","lineNumber":717,"sourceCode":"  {\n    ControllerStageTracker stageTracker = stageTrackers.get(stageId);\n    if (stageTracker == null) {\n      throw new IAE(\"Cannot find kernel corresponding to stage [%s] in query [%s]\", stageId, queryDef.getQueryId());\n    }\n    return stageTracker;\n  }\n\n  private WorkOrder getWorkOrder(int workerNumber, StageId stageId)\n  {\n    Int2ObjectMap<WorkOrder> stageWorkOrder = stageWorkOrders.get(stageId);\n\n    if (stageWorkOrder == null) {\n      throw new ISE(\"Stage[%d] work orders not found\", stageId.getStageNumber());\n    }\n\n    WorkOrder workOrder = stageWorkOrder.get(workerNumber);\n    if (workOrder == null) {\n      throw new ISE(\"Work order for worker[%d] not found for stage[%d]\", workerNumber, stageId.getStageNumber());\n    }\n    return workOrder;\n  }\n\n  /**\n   * Whether a given stage is ready to stream results to consumer stages upon transition to \"newPhase\".\n   */\n  private boolean readyToReadResults(final StageId stageId, final ControllerStagePhase newPhase)\n  {\n    if (stageOutputChannelModes.get(stageId) == OutputChannelMode.MEMORY) {\n      if (getStageDefinition(stageId).doesSortDuringShuffle()) {\n        // Sorting stages start producing output when they finish reading their input.\n        return newPhase.isDoneReadingInput();\n      } else {\n        // Non-sorting stages start producing output immediately.\n        return newPhase == ControllerStagePhase.NEW;\n      }\n    } else {","sourceCodeStart":699,"sourceCodeEnd":735,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernel.java#L699-L735","documentation":"ControllerQueryKernel.getWorkOrder looks up the WorkOrder for a specific (stage, worker) pair in the kernel's stageWorkOrders map. This ISE is thrown when the stage's work-order map exists but contains no entry for the given workerNumber, meaning the controller has no assigned work for that worker at that stage. It is an internal consistency failure: callers should only request work orders for workers that were actually allocated to the stage.","triggerScenarios":"Calling getWorkOrder(workerNumber, stageId) where stageWorkOrders.get(stageId) is non-null but stageWorkOrders.get(stageId).get(workerNumber) returns null — e.g. a worker number that was never allocated for that stage, a worker removed during retry/fault tolerance and its entry dropped, or a stage id/worker numbering mismatch (kernel recreated with different worker count).","commonSituations":"MSQ controller logs during worker retry handling (getWorkInCaseWorkerEligibleForRetry path), after partial worker failures cause reallocation, or when a controller restart restores a kernel snapshot with fewer work orders than workers referenced by runtime code.","solutions":["Verify the workerNumber is within the stage's allocated worker count (0..StageDefinition.getMaxWorkerCount()-1) before lookup.","Check whether fault tolerance/worker retry reallocated or removed the worker; use the kernel's worker-selection API instead of assuming the original worker number.","Confirm the stage id belongs to the same query definition/epoch as the kernel's stageWorkOrders map (no stale StageId after controller restart).","If reproducible without failures, capture the controller log at level DEBUG and file a Druid MSQ bug with the query id — this indicates a kernel bookkeeping bug."],"exampleFix":"// before\nWorkOrder order = kernel.getWorkOrder(workerNumber, stageId);\n\n// after\nif (workerNumber < stageDef.getMaxWorkerCount()) {\n  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);\n} else {\n  workerNumber = pickEligibleWorker(stageId);\n  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);\n}","handlingStrategy":"validation","validationCode":"if (workerNumber < 0 || workerNumber >= stageDef.getMaxWorkerCount()) {\n  throw new IllegalArgumentException(\"workerNumber out of range for stage \" + stageId);\n}","typeGuard":null,"tryCatchPattern":"try {\n  WorkOrder order = kernel.getWorkOrder(workerNumber, stageId);\n} catch (IllegalStateException e) {\n  // reassign an eligible worker or treat as kernel inconsistency\n  workerNumber = kernel.selectWorkerForStage(stageId);\n}","preventionTips":["Derive worker numbers from the stage's worker count, never from stale retry state.","After controller restart/failover, re-fetch kernel state instead of caching worker mappings.","Log worker reallocation events to correlate with this error."],"tags":["msq","controller","internal-state","worker-allocation"],"backgroundTag":"entity-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}