{"record":{"id":"0672a29a4fd1bcbd","repo":"apache/druid","slug":"worker-d-not-found-for-stage-s","errorCode":null,"errorMessage":"Worker[%d] not found for stage[%s]","messagePattern":"Worker\\[(.+?)\\] not found for stage\\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerStageTracker.java","lineNumber":292,"sourceCode":"      if (ControllerWorkerStagePhase.PRESHUFFLE_WAITING_FOR_RESULT_PARTITION_BOUNDARIES.equals(workerToPhase.get(worker))) {\n        workers.add(worker);\n      }\n    }\n    return workers;\n  }\n\n  /**\n   * Indicates that the work order for worker has been sent. Transitions the state to {@link ControllerWorkerStagePhase#READING_INPUT}\n   * if no more work orders need to be sent.\n   *\n   * @param worker\n   */\n  void workOrderSentForWorker(int worker)\n  {\n\n    workerToPhase.compute(worker, (wk, state) -> {\n      if (state == null) {\n        throw new ISE(\"Worker[%d] not found for stage[%s]\", wk, stageDef.getStageNumber());\n      }\n      if (!ControllerWorkerStagePhase.READING_INPUT.canTransitionFrom(state)) {\n        throw new ISE(\n            \"Worker[%d] cannot transistion from state[%s] to state[%s] while sending work order\",\n            worker,\n            state,\n            ControllerWorkerStagePhase.READING_INPUT\n        );\n      }\n      return ControllerWorkerStagePhase.READING_INPUT;\n    });\n    if (phase != ControllerStagePhase.READING_INPUT) {\n      if (allWorkOrdersSent()) {\n        // if no more work orders need to be sent, change state to reading input from retrying.\n        transitionTo(ControllerStagePhase.READING_INPUT);\n      }\n    }\n","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerStageTracker.java#L274-L310","documentation":"Thrown in ControllerStageTracker.workOrderSentForWorker when the given worker number has no entry in the tracker's workerToPhase map. The controller records each worker's phase when it starts the stage; reporting a work order sent for an unknown worker means the controller's worker registry and task events are out of sync.","triggerScenarios":"workOrderSentForWorker(int worker) invoked with a worker number that was never registered via initializeForWorker / stage start, e.g. duplicate or out-of-order work-order tracking, or a stale task report arriving after worker reassignment.","commonSituations":"MSQ controller crashes/restarts mid-stage with mismatched worker state; duplicate task reports from a retried worker task; custom controller extensions tracking workers incorrectly.","solutions":["Ensure every worker is registered in the stage tracker (via ControllerStagePhase initialization) before calling workOrderSentForWorker.","Guard the call with workerToPhase-style tracking or a containsWorker check.","Collect controller logs and the query id and file a Druid MSQ issue if this occurs in unmodified core code."],"exampleFix":"// before\ntracker.workOrderSentForWorker(worker); // throws if unknown\n// after\nif (tracker.getWorkers().contains(worker)) {\n  tracker.workOrderSentForWorker(worker);\n}","handlingStrategy":"validation","validationCode":"if (tracker.getWorkerCount() > 0 && knownWorkers.contains(worker)) {\n  tracker.workOrderSentForWorker(worker);\n}","typeGuard":null,"tryCatchPattern":"try {\n  tracker.workOrderSentForWorker(worker);\n} catch (IllegalStateException e) {\n  LOG.warn(e, \"Ignoring work order ack for unknown worker %d\", worker);\n}","preventionTips":["Register workers in the stage tracker before emitting work orders.","Make work-order handling idempotent and ordered per task id."],"tags":["druid","msq","controller","worker-tracking"],"backgroundTag":"record-not-found","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}