apache/druid · critical · MSQException

TooManyAttemptsForWorker

Error message

TooManyAttemptsForWorker

What it means

MSQWorkerTaskLauncher.checkRelaunchLimitsOrThrow throws MSQException(TooManyAttemptsForWorker) when a worker task's retry count exceeds Limits.PER_WORKER_RELAUNCH_LIMIT, meaning the same worker task kept crashing/failing and relaunching beyond the allowed attempts. The fault includes the limit, task id, worker number, and the last error message from the failed attempt.

Solutions

1. Read the errorMsg in TooManyAttemptsForWorker and fix the underlying worker crash cause (usually OOM or config error)
2. Remove/quarantine the unhealthy middle manager node and rerun the query
3. Increase worker container memory or lower the task JVM heap so the process stops being killed
4. Adjust retry/relaunch limits only after fixing the root cause; re-running on a healthy cluster usually succeeds

Example fix

// before: task JVM -Xmx6g inside a 4GB container => repeated OOM kills
// after
// -Xmx3g (or raise the container to 8g), then rerun the query

// ensure container memory exceeds task heap to avoid OOM-kill relaunch loops
if (containerMemoryBytes <= taskHeapBytes + overhead) {
  throw new IllegalStateException("Container memory too small for task heap");
}

try {
  runMsqQuery();
} catch (MSQException e) {
  if (e.getFault() instanceof TooManyAttemptsForWorker f) {
    log.error("Worker %s failed %d times: %s", f.getWorkerNumber(), f.getLimit(), f.getErrorMsg());
    // fix node/config, then resubmit
  } else { throw e; }
}

When it happens

Trigger: A worker task crashes on startup repeatedly (bad JVM args, OOM kill, corrupt local state) so the launcher keeps relaunching it until the per-worker relaunch limit is exceeded; persistent node-specific failure (bad disk, kernel OOM killer).

Common situations: Peon tasks repeatedly OOM-killed by the OS because heap is larger than container memory; one unhealthy middle manager node; local temp dir full or unwritable causing immediate task death loops.

Related errors

• Broadcast input number out of range
• BroadcastTablesTooLarge
• Can not supply empty segments as input, please use either…
• CanceledFault(CancellationReason.UNKNOWN)
• Cannot accept duplicate stage numbers

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/1ef5fc99ac1969eb. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java:772

        synchronized (taskIds) {
          // replace taskId with the retry taskID for the same worker number
          taskIdToWorkerNumber.remove(taskIds.get(toRelaunch.getWorkerNumber()));
          taskIds.set(toRelaunch.getWorkerNumber(), relaunchedTask.getId());
          taskIdToWorkerNumber.put(relaunchedTask.getId(), toRelaunch.getWorkerNumber());
          taskIds.notifyAll();
        }

        return taskHistory;
      });
      // remove the worker from the relaunch set
      iterator.remove();
    }
  }

  private void checkRelaunchLimitsOrThrow(TaskTracker tracker, MSQWorkerTask relaunchTask)
  {
    if (relaunchTask.getRetryCount() > Limits.PER_WORKER_RELAUNCH_LIMIT) {
      throw new MSQException(new TooManyAttemptsForWorker(
          Limits.PER_WORKER_RELAUNCH_LIMIT,
          relaunchTask.getId(),
          relaunchTask.getWorkerNumber(),
          tracker.statusRef.get().getErrorMsg()
      ));
    }
    if (currentRelaunchCount > Limits.TOTAL_RELAUNCH_LIMIT) {
      throw new MSQException(new TooManyAttemptsForJob(
          Limits.TOTAL_RELAUNCH_LIMIT,
          currentRelaunchCount,
          relaunchTask.getId(),
          tracker.statusRef.get().getErrorMsg()
      ));
    }
  }

  private void shutDownTasks()
  {

View on GitHub (pinned to 9b90983fd2)