{"record":{"id":"4b8d92a7ff07b260","repo":"apache/druid","slug":"exception-while-getting-active-tasks-from-overlord","errorCode":null,"errorMessage":"Exception while getting active tasks from Overlord. Will retry on next scheduled run.","messagePattern":"Exception while getting active tasks from Overlord\\. Will retry on next scheduled run\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"indexing-service/src/main/java/org/apache/druid/indexing/worker/WorkerTaskManager.java","lineNumber":648,"sourceCode":"    }\n    catch (ExecutionException e) {\n      if (e.getCause() instanceof HttpResponseException) {\n        final HttpResponseStatus status = ((HttpResponseException) e.getCause()).getResponse().getStatus();\n        if (status.getCode() == 404) {\n          // NOTE: this is to support backward compatibility, when overlord doesn't have \"activeTasks\" endpoint.\n          // this if clause should be removed in a future release.\n          log.debug(\"Deleting all completed tasks. Overlord appears to be running on older version.\");\n          apiCallResult = Either.value(ImmutableMap.of());\n        } else {\n          apiCallResult = Either.error(e.getCause());\n        }\n      } else {\n        apiCallResult = Either.error(e.getCause());\n      }\n    }\n\n    if (apiCallResult.isError()) {\n      log.warn(\n          apiCallResult.error(),\n          \"Exception while getting active tasks from Overlord. Will retry on next scheduled run.\"\n      );\n\n      return;\n    }\n\n    for (String taskId : taskIds) {\n      TaskStatus status = apiCallResult.valueOrThrow().get(taskId);\n      if (status == null || status.isComplete()) {\n        log.debug(\n            \"Deleting completed task[%s] information, Overlord task status[%s].\",\n            taskId,\n            status == null ? \"unknown\" : status.getStatusCode()\n        );\n\n        completedTasks.remove(taskId);\n        File taskFile = new File(getCompletedTaskDir(), taskId);","sourceCodeStart":630,"sourceCodeEnd":666,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/indexing-service/src/main/java/org/apache/druid/indexing/worker/WorkerTaskManager.java#L630-L666","documentation":"WorkerTaskManager.doCompletedTasksCleanup calls the Overlord API to fetch active tasks before cleaning up completed tasks locally. If that HTTP call fails (apiCallResult.isError()), the manager logs a warning and returns, deferring cleanup to the next scheduled run. No tasks are removed in this cycle; cleanup is idempotent and retried periodically.","triggerScenarios":"The periodic completed-tasks cleanup job calls the Overlord's active-task endpoint and the request fails: Overlord is down/restarting, the HTTP request times out, TLS or auth fails, or the network path between the worker and Overlord is broken.","commonSituations":"Transient Overlord restarts or leader elections, network partitions between middleManager workers and the Overlord, or misconfigured druid.host / overlord discovery causing connection failures.","solutions":["Check the attached exception in the worker logs to see whether the failure is connection-refused, timeout, or auth-related.","Verify the Overlord is reachable from the worker: confirm druid.host, discovery config, and any proxy/firewall rules.","Inspect Overlord health; restarts during leader election commonly cause transient failures that self-heal on the next scheduled run."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  workerTaskManager.cleanupCompletedTasks();\n} catch (Exception e) {\n  log.warn(e, \"Overlord unreachable; completed-tasks cleanup will retry on next scheduled run\");\n}","preventionTips":["Monitor Overlord availability and leader-election events; cleanup retries are transient-safe.","Verify druid.host and discovery configuration on workers so the Overlord API is reachable.","Watch for network/firewall changes between middleManager workers and the Overlord."],"tags":["druid","overlord","http","worker","network"],"backgroundTag":"http-request-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}