{"record":{"id":"f5c109257717f2a1","repo":"apache/druid","slug":"worker-completion-callback-error-for-stage-s","errorCode":null,"errorMessage":"Worker completion callback error for stage [%s]","messagePattern":"Worker completion callback error for stage \\[(.+?)\\]","errorType":"exception","errorClass":"RE","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java","lineNumber":783,"sourceCode":"        ReadableFrameChannel readableChannel = null;\n\n        try {\n          readableChannel = channel.getReadableChannel();\n          getOrCreateStageOutputHolder(stageId, channel.getPartitionNumber())\n              .setChannel(readableChannel);\n        }\n        catch (Exception e) {\n          if (readableChannel != null) {\n            try {\n              readableChannel.close();\n            }\n            catch (Throwable e2) {\n              e.addSuppressed(e2);\n            }\n          }\n\n          kernelManipulationQueue.add(holder -> {\n            throw new RE(e, \"Worker completion callback error for stage [%s]\", stageId);\n          });\n        }\n      }\n\n      @Override\n      public void onSuccess(Object resultObject)\n      {\n        kernelManipulationQueue.add(\n            holder -> {\n              // Call finishProcessing prior to transitioning to RESULTS_COMPLETE, so the FrameContext is closed\n              // and resources are released.\n              holder.finishProcessing(stageId);\n\n              final WorkerStageKernel kernel = holder.getKernelFor(stageId);\n              if (kernel != null) {\n                kernel.setResultsComplete(resultObject);\n              }\n            }","sourceCodeStart":765,"sourceCodeEnd":801,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java#L765-L801","documentation":"In WorkerImpl.onOutputChannelAvailable, when the worker-completion callback for a stage fails, the original throwable is rethrown inside the kernel manipulation queue wrapped in a RuntimeException with message \"Worker completion callback error for stage [%s]\". It means a worker finished its stage output but the completion callback (e.g. notifying the controller) threw, poisoning the kernel queue.","triggerScenarios":"The onSuccess/onFailure completion callback path for a stage's output channel throws any Throwable; the catch block adds suppressed exceptions and injects a throwing kernel, producing this RE with the failing stage id.","commonSituations":"Controller is unreachable or restarted while worker completes a stage; network partition between peon and controller during callback; serialization error in the result object delivered to the callback; bug in a custom listener.","solutions":["Inspect the full stack trace and its suppressed exceptions to find the root cause (usually the callback's underlying IOException/HTTP error)","Verify controller availability and network connectivity from the peon/task logs","Retry the query; transient controller unavailability is the most common cause","If reproducible, report/inspect the stage callback code path in WorkerImpl for framework bugs"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// pre-check controller reachability before submitting\nboolean controllerUp = ping(\"http://\" + coordinatorHost + \":\" + coordinatorPort + \"/status\");","typeGuard":null,"tryCatchPattern":"try {\n  runMsqQuery();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Worker completion callback error\")) {\n    // inspect suppressed exceptions, retry after controller recovers\n    retryWithBackoff();\n  } else { throw e; }\n}","preventionTips":["Keep the controller task alive until all workers report completion","Monitor peon→controller network paths in the cluster","Upgrade Druid if the root cause is a known framework bug in the completion callback"],"tags":["msq","callback","controller-communication"],"backgroundTag":"callback-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}