{"record":{"id":"a3e2f6fc1f9d9190","repo":"apache/druid","slug":"taskstarttimeoutfault","errorCode":null,"errorMessage":"TaskStartTimeoutFault","messagePattern":"TaskStartTimeoutFault","errorType":"error_code","errorClass":"MSQException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java","lineNumber":652,"sourceCode":"   * have gone inexplicably missing.\n   * <p>\n   * Throws an exception if some task is erroneous.\n   */\n  private void checkForErroneousTasks()\n  {\n    final int numTasks = taskTrackers.size();\n\n    for (Map.Entry<String, TaskTracker> taskEntry : taskTrackersByWorkerNumber()) {\n      final String taskId = taskEntry.getKey();\n      final TaskTracker tracker = taskEntry.getValue();\n      if (tracker.isRetryCandidate()) {\n        continue;\n      }\n      if (tracker.statusRef.get() != null\n          && tracker.didRunTimeOut(maxTaskStartDelayMillis)\n          && !canceledWorkerTasks.contains(taskId)) {\n        removeWorkerFromFullyStartedWorkers(tracker);\n        throw new MSQException(new TaskStartTimeoutFault(\n            this.getWorkerCount().getPendingWorkerCount(),\n            numTasks + 1,\n            maxTaskStartDelayMillis\n        ));\n      } else if (tracker.statusRef.get() == null || (tracker.didFail() && !canceledWorkerTasks.contains(taskId))) {\n        startRetryingTasksIfNeeded(tracker, taskId);\n      }\n    }\n  }\n\n  private void startRetryingTasksIfNeeded(TaskTracker tracker, String taskId)\n  {\n    tracker.enableRetry();\n    removeWorkerFromFullyStartedWorkers(tracker);\n    MSQFault msqFault = generateFailureFault(taskId, tracker.statusRef.get());\n    log.info(\"Task[%s] failed caused of [%s]. Trying to relaunch the worker\", taskId, msqFault);\n    invokeFailureListener(tracker, msqFault);\n  }","sourceCodeStart":634,"sourceCodeEnd":670,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java#L634-L670","documentation":"MSQWorkerTaskLauncher.checkForErroneousTasks throws MSQException(TaskStartTimeoutFault) when a worker task's status has not appeared within maxTaskStartDelayMillis (druid.msq.indexing.worker.taskStartDelay property / configured timeout), meaning the overlord failed to launch a worker task in time. The fault reports pending worker count, required numTasks+1, and the delay used.","triggerScenarios":"Overlord saturated or slow to schedule tasks; middleManager/worker pool full so pending tasks queue past the start-delay timeout; cluster capacity insufficient for the requested number of MSQ workers; overlord restart during task launch.","commonSituations":"Large MSQ queries requesting more tasks than the cluster can concurrently run; indexer/middleManager autoscaling lagging; paused or misconfigured worker groups; overlord backlogs after restarts.","solutions":["Scale out middle managers/indexers or raise their task slot capacity so pending workers start within the timeout","Increase druid.msq.indexing.worker.taskStartDelay (task start timeout) if the cluster is just slow","Reduce maxNumTasks in the query context to fit available cluster capacity","Check overlord logs/metrics for scheduling backlog and fix the root cause (e.g. disabled worker groups, autoscaler issues)"],"exampleFix":"// before: default 15s start delay on a saturated cluster\n// after: raise the timeout and reduce task count\n// runtime.properties: druid.msq.indexing.worker.taskStartDelay=PT5M\n// query context: {\"maxNumTasks\": 8}","handlingStrategy":"validation","validationCode":"int capacity = overlord.getAvailableTaskSlots();\nint requested = queryContext.getMaxNumTasks();\nif (requested > capacity) {\n  throw new IllegalStateException(\"Requested \" + requested + \" tasks but only \" + capacity + \" slots available\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  runMsqQuery();\n} catch (MSQException e) {\n  if (e.getFault() instanceof TaskStartTimeoutFault f) {\n    // resubmit with smaller maxNumTasks or after scaling the worker pool\n  } else { throw e; }\n}","preventionTips":["Right-size maxNumTasks to actual cluster capacity","Provision enough middle manager/indexer slots for peak concurrency","Raise worker taskStartDelay on slow-scheduling clusters"],"tags":["msq","task-launch","timeout","overlord"],"backgroundTag":"task-start-timeout","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}