{"record":{"id":"1ef5fc99ac1969eb","repo":"apache/druid","slug":"toomanyattemptsforworker","errorCode":null,"errorMessage":"TooManyAttemptsForWorker","messagePattern":"TooManyAttemptsForWorker","errorType":"error_code","errorClass":"MSQException","httpStatus":null,"severity":"critical","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java","lineNumber":772,"sourceCode":"        synchronized (taskIds) {\n          // replace taskId with the retry taskID for the same worker number\n          taskIdToWorkerNumber.remove(taskIds.get(toRelaunch.getWorkerNumber()));\n          taskIds.set(toRelaunch.getWorkerNumber(), relaunchedTask.getId());\n          taskIdToWorkerNumber.put(relaunchedTask.getId(), toRelaunch.getWorkerNumber());\n          taskIds.notifyAll();\n        }\n\n        return taskHistory;\n      });\n      // remove the worker from the relaunch set\n      iterator.remove();\n    }\n  }\n\n  private void checkRelaunchLimitsOrThrow(TaskTracker tracker, MSQWorkerTask relaunchTask)\n  {\n    if (relaunchTask.getRetryCount() > Limits.PER_WORKER_RELAUNCH_LIMIT) {\n      throw new MSQException(new TooManyAttemptsForWorker(\n          Limits.PER_WORKER_RELAUNCH_LIMIT,\n          relaunchTask.getId(),\n          relaunchTask.getWorkerNumber(),\n          tracker.statusRef.get().getErrorMsg()\n      ));\n    }\n    if (currentRelaunchCount > Limits.TOTAL_RELAUNCH_LIMIT) {\n      throw new MSQException(new TooManyAttemptsForJob(\n          Limits.TOTAL_RELAUNCH_LIMIT,\n          currentRelaunchCount,\n          relaunchTask.getId(),\n          tracker.statusRef.get().getErrorMsg()\n      ));\n    }\n  }\n\n  private void shutDownTasks()\n  {","sourceCodeStart":754,"sourceCodeEnd":790,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/indexing/MSQWorkerTaskLauncher.java#L754-L790","documentation":"MSQWorkerTaskLauncher.checkRelaunchLimitsOrThrow throws MSQException(TooManyAttemptsForWorker) when a worker task's retry count exceeds Limits.PER_WORKER_RELAUNCH_LIMIT, meaning the same worker task kept crashing/failing and relaunching beyond the allowed attempts. The fault includes the limit, task id, worker number, and the last error message from the failed attempt.","triggerScenarios":"A worker task crashes on startup repeatedly (bad JVM args, OOM kill, corrupt local state) so the launcher keeps relaunching it until the per-worker relaunch limit is exceeded; persistent node-specific failure (bad disk, kernel OOM killer).","commonSituations":"Peon tasks repeatedly OOM-killed by the OS because heap is larger than container memory; one unhealthy middle manager node; local temp dir full or unwritable causing immediate task death loops.","solutions":["Read the errorMsg in TooManyAttemptsForWorker and fix the underlying worker crash cause (usually OOM or config error)","Remove/quarantine the unhealthy middle manager node and rerun the query","Increase worker container memory or lower the task JVM heap so the process stops being killed","Adjust retry/relaunch limits only after fixing the root cause; re-running on a healthy cluster usually succeeds"],"exampleFix":"// before: task JVM -Xmx6g inside a 4GB container => repeated OOM kills\n// after\n// -Xmx3g (or raise the container to 8g), then rerun the query","handlingStrategy":"retry","validationCode":"// ensure container memory exceeds task heap to avoid OOM-kill relaunch loops\nif (containerMemoryBytes <= taskHeapBytes + overhead) {\n  throw new IllegalStateException(\"Container memory too small for task heap\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  runMsqQuery();\n} catch (MSQException e) {\n  if (e.getFault() instanceof TooManyAttemptsForWorker f) {\n    log.error(\"Worker %s failed %d times: %s\", f.getWorkerNumber(), f.getLimit(), f.getErrorMsg());\n    // fix node/config, then resubmit\n  } else { throw e; }\n}","preventionTips":["Set task heap well below container memory limits","Watch for nodes with repeated task failures and drain them","Fix the root-cause errorMsg before simply raising relaunch limits"],"tags":["msq","task-retry","worker-crash","relaunch-limit"],"backgroundTag":"retry-limit-exceeded","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}