apache/druid · error · MSQException

TaskStartTimeoutFault

Error message

TaskStartTimeoutFault

What it means

MSQWorkerTaskLauncher.checkForErroneousTasks throws MSQException(TaskStartTimeoutFault) when a worker task's status has not appeared within maxTaskStartDelayMillis (druid.msq.indexing.worker.taskStartDelay property / configured timeout), meaning the overlord failed to launch a worker task in time. The fault reports pending worker count, required numTasks+1, and the delay used.

Solutions

1. Scale out middle managers/indexers or raise their task slot capacity so pending workers start within the timeout
2. Increase druid.msq.indexing.worker.taskStartDelay (task start timeout) if the cluster is just slow
3. Reduce maxNumTasks in the query context to fit available cluster capacity
4. Check overlord logs/metrics for scheduling backlog and fix the root cause (e.g. disabled worker groups, autoscaler issues)

Example fix

// before: default 15s start delay on a saturated cluster
// after: raise the timeout and reduce task count
// runtime.properties: druid.msq.indexing.worker.taskStartDelay=PT5M
// query context: {"maxNumTasks": 8}

int capacity = overlord.getAvailableTaskSlots();
int requested = queryContext.getMaxNumTasks();
if (requested > capacity) {
  throw new IllegalStateException("Requested " + requested + " tasks but only " + capacity + " slots available");
}

try {
  runMsqQuery();
} catch (MSQException e) {
  if (e.getFault() instanceof TaskStartTimeoutFault f) {
    // resubmit with smaller maxNumTasks or after scaling the worker pool
  } else { throw e; }
}

When it happens

Trigger: Overlord saturated or slow to schedule tasks; middleManager/worker pool full so pending tasks queue past the start-delay timeout; cluster capacity insufficient for the requested number of MSQ workers; overlord restart during task launch.

Common situations: Large MSQ queries requesting more tasks than the cluster can concurrently run; indexer/middleManager autoscaling lagging; paused or misconfigured worker groups; overlord backlogs after restarts.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Access-Check-Result
• Action [ ] failed for worker [ ] with status ( )
• At least one task runner must be enabled
• authResult.getErrorMessage()
• Batched segment allocation is disabled

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a3e2f6fc1f9d9190. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java:652

   * have gone inexplicably missing.
   * <p>
   * Throws an exception if some task is erroneous.
   */
  private void checkForErroneousTasks()
  {
    final int numTasks = taskTrackers.size();

    for (Map.Entry<String, TaskTracker> taskEntry : taskTrackersByWorkerNumber()) {
      final String taskId = taskEntry.getKey();
      final TaskTracker tracker = taskEntry.getValue();
      if (tracker.isRetryCandidate()) {
        continue;
      }
      if (tracker.statusRef.get() != null
          && tracker.didRunTimeOut(maxTaskStartDelayMillis)
          && !canceledWorkerTasks.contains(taskId)) {
        removeWorkerFromFullyStartedWorkers(tracker);
        throw new MSQException(new TaskStartTimeoutFault(
            this.getWorkerCount().getPendingWorkerCount(),
            numTasks + 1,
            maxTaskStartDelayMillis
        ));
      } else if (tracker.statusRef.get() == null || (tracker.didFail() && !canceledWorkerTasks.contains(taskId))) {
        startRetryingTasksIfNeeded(tracker, taskId);
      }
    }
  }

  private void startRetryingTasksIfNeeded(TaskTracker tracker, String taskId)
  {
    tracker.enableRetry();
    removeWorkerFromFullyStartedWorkers(tracker);
    MSQFault msqFault = generateFailureFault(taskId, tracker.statusRef.get());
    log.info("Task[%s] failed caused of [%s]. Trying to relaunch the worker", taskId, msqFault);
    invokeFailureListener(tracker, msqFault);
  }

View on GitHub (pinned to 9b90983fd2)