apache/dolphinscheduler · error · TaskDispatchException

"Dispatch task: " + taskName + " to " + physicalTaskExecutor

Error message

"Dispatch task: " + taskName + " to " + physicalTaskExecutorAddress + " failed: " + taskExecutorDispatchResponse

What it means

The master selected a worker and sent the dispatch RPC, but the worker's IPhysicalTaskExecutorOperator.dispatchTask returned a TaskExecutorDispatchResponse that is not successful, so a TaskDispatchException is thrown including the worker address and response details. Unlike error 613, the RPC completed — the worker explicitly reported the dispatch as failed.

Source

Thrown at dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/task/client/PhysicalTaskExecutorClientDelegator.java:91

        }

        // select an available worker from the worker group; throws NoAvailableWorkerException if none is available.
        final String physicalTaskExecutorAddress = workerLoadBalancer
                .select(workerGroup)
                .map(Host::of)
                .map(Host::getAddress)
                .orElseThrow(() -> new NoAvailableWorkerException(workerGroup));

        taskExecutionContext.setHost(physicalTaskExecutorAddress);
        taskExecution.getTaskInstance().setHost(physicalTaskExecutorAddress);

        try {
            final TaskExecutorDispatchResponse taskExecutorDispatchResponse = Clients
                    .withService(IPhysicalTaskExecutorOperator.class)
                    .withHost(physicalTaskExecutorAddress)
                    .dispatchTask(TaskExecutorDispatchRequest.of(taskExecution.getTaskExecutionContext()));
            if (!taskExecutorDispatchResponse.isDispatchSuccess()) {
                throw new TaskDispatchException(
                        "Dispatch task: " + taskName + " to " + physicalTaskExecutorAddress + " failed: "
                                + taskExecutorDispatchResponse);
            }
        } catch (TaskDispatchException e) {
            throw e;
        } catch (Exception e) {
            throw new TaskDispatchException(
                    "Dispatch task: " + taskName + " to " + physicalTaskExecutorAddress + " failed", e);
        }
    }

    @Override
    public boolean reassignMasterHost(final ITaskExecution taskExecution) {
        final String taskName = taskExecution.getName();
        checkArgument(taskExecution.isTaskInstanceInitialized(),
                "Task " + taskName + "is not initialized cannot take-over");

        final TaskInstance taskInstance = taskExecution.getTaskInstance();

View on GitHub (pinned to 02eac45a1b)

Solutions

  1. Read the appended TaskExecutorDispatchResponse in the message for the worker's failure reason and fix that cause
  2. Confirm the worker has the task plugin installed and its version matches the master
  3. Check worker logs and resources (memory, disk, task-exec threads) around the failure timestamp
  4. Retry the task/workflow instance once the worker is healthy; consider draining/restarting the problematic worker

Example fix

// before
// failure only visible after a failed dispatch round-trip
// after
// keep worker groups healthy and verified before submitting work
if (!workerClusters.containsWorkerGroup(workerGroup)) {
    throw new WorkerGroupNotFoundException(workerGroup); // fail fast instead of a worker-side rejection
}
Defensive patterns

Strategy: retry

Validate before calling

// keep the worker group verified before dispatch
if (!clusterManager.getWorkerClusters().containsWorkerGroup(workerGroup)) {
    throw new WorkerGroupNotFoundException(workerGroup);
}

Type guard

boolean dispatchSucceeded(TaskExecutorDispatchResponse r) { return r != null && r.isDispatchSuccess(); }

Try / catch

try {
    taskExecutorClient.dispatch(taskExecution);
} catch (TaskDispatchException e) {
    log.warn("Dispatch to worker failed ({}), scheduling retry", e.getMessage());
    // re-enqueue via workflow retry strategy
}

Prevention

When it happens

Trigger: Clients.withService(IPhysicalTaskExecutorOperator.class).withHost(address).dispatchTask(TaskExecutorDispatchRequest.of(ctx)) succeeds at the transport level, but the response's isDispatchSuccess() is false.

Common situations: Worker rejects the task because its task plugin for that type is missing or incompatible (master/worker version skew); worker out of memory/disk or thread pool exhausted; task resource (jar/script) download fails on the worker; worker in a bad state right after startup or during shutdown.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/5b59eae785059832. Report an issue: GitHub.