apache/druid · error · MSQException

TooManyAttemptsForJob

Error message

TooManyAttemptsForJob

What it means

A guard exception raised in checkRelaunchLimitsOrThrow when a worker task's retry (relaunch) count exceeds Limits.PER_WORKER_RELAUNCH_LIMIT. The controller relaunches failed workers only a bounded number of times; once a single worker exceeds that budget the launcher gives up on it and surfaces TooManyAttemptsForWorker so the query fails fast instead of looping forever.

Solutions

1. Inspect the worker task logs and tracker.getErrorMsg() to find the root cause of the repeated failures (OOM, node loss, task queue overflow) and fix that first.
2. Increase Limits.PER_WORKER_RELAUNCH_LIMIT / TOTAL_RELAUNCH_LIMIT via MSQ tuning config if transient infrastructure instability is expected.
3. Re-run the query on a healthier cluster; persistent worker failures are usually environmental (spot interruptions, resource exhaustion).

When it happens

Trigger: Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java:780 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/fc4983248c23ee95. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java:780

        return taskHistory;
      });
      // remove the worker from the relaunch set
      iterator.remove();
    }
  }

  private void checkRelaunchLimitsOrThrow(TaskTracker tracker, MSQWorkerTask relaunchTask)
  {
    if (relaunchTask.getRetryCount() > Limits.PER_WORKER_RELAUNCH_LIMIT) {
      throw new MSQException(new TooManyAttemptsForWorker(
          Limits.PER_WORKER_RELAUNCH_LIMIT,
          relaunchTask.getId(),
          relaunchTask.getWorkerNumber(),
          tracker.statusRef.get().getErrorMsg()
      ));
    }
    if (currentRelaunchCount > Limits.TOTAL_RELAUNCH_LIMIT) {
      throw new MSQException(new TooManyAttemptsForJob(
          Limits.TOTAL_RELAUNCH_LIMIT,
          currentRelaunchCount,
          relaunchTask.getId(),
          tracker.statusRef.get().getErrorMsg()
      ));
    }
  }

  private void shutDownTasks()
  {

    cleanFailedTasksWhichAreRelaunched();
    for (final Map.Entry<String, TaskTracker> taskEntry : taskTrackersByWorkerNumber()) {
      final String taskId = taskEntry.getKey();
      final TaskTracker tracker = taskEntry.getValue();
      if ((!canceledWorkerTasks.contains(taskId))
          &&
          (!tracker.isComplete())) {

View on GitHub (pinned to 9b90983fd2)