apache/dolphinscheduler · error · TaskDispatchException

"Dispatch task: " + taskName + " to " + physicalTaskExecutor

Error message

"Dispatch task: " + taskName + " to " + physicalTaskExecutorAddress + " failed"

What it means

A generic wrapper: any exception thrown while sending the dispatch RPC to the selected worker (network error, timeout, serialization failure, unexpected runtime exception) is rethrown as TaskDispatchException with message 'Dispatch task: <name> to <address> failed' and the original exception as the cause. The dispatch never got a usable response from the worker.

Source

Thrown at dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/task/client/PhysicalTaskExecutorClientDelegator.java:98

                .orElseThrow(() -> new NoAvailableWorkerException(workerGroup));

        taskExecutionContext.setHost(physicalTaskExecutorAddress);
        taskExecution.getTaskInstance().setHost(physicalTaskExecutorAddress);

        try {
            final TaskExecutorDispatchResponse taskExecutorDispatchResponse = Clients
                    .withService(IPhysicalTaskExecutorOperator.class)
                    .withHost(physicalTaskExecutorAddress)
                    .dispatchTask(TaskExecutorDispatchRequest.of(taskExecution.getTaskExecutionContext()));
            if (!taskExecutorDispatchResponse.isDispatchSuccess()) {
                throw new TaskDispatchException(
                        "Dispatch task: " + taskName + " to " + physicalTaskExecutorAddress + " failed: "
                                + taskExecutorDispatchResponse);
            }
        } catch (TaskDispatchException e) {
            throw e;
        } catch (Exception e) {
            throw new TaskDispatchException(
                    "Dispatch task: " + taskName + " to " + physicalTaskExecutorAddress + " failed", e);
        }
    }

    @Override
    public boolean reassignMasterHost(final ITaskExecution taskExecution) {
        final String taskName = taskExecution.getName();
        checkArgument(taskExecution.isTaskInstanceInitialized(),
                "Task " + taskName + "is not initialized cannot take-over");

        final TaskInstance taskInstance = taskExecution.getTaskInstance();
        final String taskExecutorHost = taskInstance.getHost();
        if (StringUtils.isEmpty(taskExecutorHost)) {
            log.debug(
                    "The task executor: {} host is empty, cannot take-over, this might caused by the task hasn't dispatched",
                    taskName);
            return false;
        }

View on GitHub (pinned to 02eac45a1b)

Solutions

Inspect the caused-by exception in the log to identify the transport failure (connection refused vs timeout vs unmarshal)
Check the target worker at the address in the message: is it running, reachable, and registered with the correct host:port?
Verify network/firewall rules allow master->worker RPC on the worker's listen port
Retry the task; if a specific worker is persistently bad, remove it from the group or fix its advertised address

Example fix

// before
// raw RPC exception bubbles as generic TaskDispatchException
// after
// validate reachability before dispatch
if (!isWorkerReachable(physicalTaskExecutorAddress)) {
    throw new NoAvailableWorkerException(workerGroup);
}

Defensive patterns

Strategy: try-catch

Try / catch

try {
    taskExecutorClient.dispatch(taskExecution);
} catch (TaskDispatchException e) {
    Throwable cause = e.getCause();
    log.error("Dispatch transport failure to {}: {}", cause, e.getMessage());
    // if cause is a connectivity error, mark the worker unhealthy and retry on another worker
}

Prevention

Monitor master<->worker connectivity (ports, firewalls) continuously
Ensure workers advertise the correct host:port in the registry
Set sane RPC timeouts and worker saturation limits
Use retry/failover so a single dead worker does not fail the workflow

When it happens

Trigger: Any Exception other than TaskDispatchException escaping the Clients...dispatchTask call in PhysicalTaskExecutorClientDelegator.dispatch — e.g. connection refused/reset, RPC timeout, unmarshalling error — caught by the catch (Exception e) block.

Common situations: Worker process crashed or was restarted mid-dispatch; firewall/network partition between master and worker; wrong host:port advertised by the worker in the registry; RPC timeout because the worker is saturated.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/1ab5e4d664efe047. Report an issue: GitHub.