What causes "Worker completion callback error for stage [%s]" in apache/druid?

Inspect the full stack trace and its suppressed exceptions to find the root cause (usually the callback's underlying IOException/HTTP error)

apache/druid · error · RE

Worker completion callback error for stage

Error message

Worker completion callback error for stage [%s]

What it means

In WorkerImpl.onOutputChannelAvailable, when the worker-completion callback for a stage fails, the original throwable is rethrown inside the kernel manipulation queue wrapped in a RuntimeException with message "Worker completion callback error for stage [%s]". It means a worker finished its stage output but the completion callback (e.g. notifying the controller) threw, poisoning the kernel queue.

Solutions

Inspect the full stack trace and its suppressed exceptions to find the root cause (usually the callback's underlying IOException/HTTP error)
Verify controller availability and network connectivity from the peon/task logs
Retry the query; transient controller unavailability is the most common cause
If reproducible, report/inspect the stage callback code path in WorkerImpl for framework bugs

Defensive patterns

Strategy: retry

Validate before calling

// pre-check controller reachability before submitting
boolean controllerUp = ping("http://" + coordinatorHost + ":" + coordinatorPort + "/status");

Try / catch

try {
  runMsqQuery();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Worker completion callback error")) {
    // inspect suppressed exceptions, retry after controller recovers
    retryWithBackoff();
  } else { throw e; }
}

Prevention

Keep the controller task alive until all workers report completion
Monitor peon→controller network paths in the cluster
Upgrade Druid if the root cause is a known framework bug in the completion callback

When it happens

Trigger: The onSuccess/onFailure completion callback path for a stage's output channel throws any Throwable; the catch block adds suppressed exceptions and injects a throwing kernel, producing this RE with the failing stage id.

Common situations: Controller is unreachable or restarted while worker completes a stage; network partition between peon and controller during callback; serialization error in the result object delivered to the callback; bug in a custom listener.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f5c109257717f2a1. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java:783

        ReadableFrameChannel readableChannel = null;

        try {
          readableChannel = channel.getReadableChannel();
          getOrCreateStageOutputHolder(stageId, channel.getPartitionNumber())
              .setChannel(readableChannel);
        }
        catch (Exception e) {
          if (readableChannel != null) {
            try {
              readableChannel.close();
            }
            catch (Throwable e2) {
              e.addSuppressed(e2);
            }
          }

          kernelManipulationQueue.add(holder -> {
            throw new RE(e, "Worker completion callback error for stage [%s]", stageId);
          });
        }
      }

      @Override
      public void onSuccess(Object resultObject)
      {
        kernelManipulationQueue.add(
            holder -> {
              // Call finishProcessing prior to transitioning to RESULTS_COMPLETE, so the FrameContext is closed
              // and resources are released.
              holder.finishProcessing(stageId);

              final WorkerStageKernel kernel = holder.getKernelFor(stageId);
              if (kernel != null) {
                kernel.setResultsComplete(resultObject);
              }
            }

View on GitHub (pinned to 9b90983fd2)