apache/dolphinscheduler · error · TaskKillException

"Kill task: " + taskExecution.getName() + " from executor fa

Error message

"Kill task: " + taskExecution.getName() + " from executor failed"

What it means

Thrown as TaskKillException when a kill request for a task cannot be completed on its executor. TaskExecutorClient.kill delegates to the executor client delegator; any exception in the RPC is wrapped with the task name. It means the master could not force-stop the task remotely.

Source

Thrown at dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/task/client/TaskExecutorClient.java:90

        }
    }

    @Override
    public void pause(final ITaskExecution taskExecution) throws TaskPauseException {
        try {
            getTaskExecutorClientDelegator(taskExecution).pause(taskExecution);
        } catch (Exception ex) {
            throw new TaskPauseException("Pause task: " + taskExecution.getName() + " from executor failed",
                    ex);
        }
    }

    @Override
    public void kill(final ITaskExecution taskExecution) throws TaskKillException {
        try {
            getTaskExecutorClientDelegator(taskExecution).kill(taskExecution);
        } catch (Exception ex) {
            throw new TaskKillException("Kill task: " + taskExecution.getName() + " from executor failed", ex);
        }
    }

    @Override
    public void ackTaskExecutorLifecycleEvent(
                                              final ITaskExecution taskExecution,
                                              final ITaskExecutorLifecycleEventReporter.TaskExecutorLifecycleEventAck taskExecutorLifecycleEventAck) {
        try {
            if (StringUtils.isEmpty(taskExecution.getTaskInstance().getHost())) {
                log.info("The task: {} is didn't dispatched to executor, skip ack taskExecutorLifecycleEventAck: {}",
                        taskExecution.getName(), taskExecutorLifecycleEventAck);
                return;
            }
            getTaskExecutorClientDelegator(taskExecution)
                    .ackTaskExecutorLifecycleEvent(taskExecution, taskExecutorLifecycleEventAck);
        } catch (Exception ex) {
            log.error("Send taskExecutorLifecycleEventAck: {} failed", taskExecutorLifecycleEventAck, ex);
        }

View on GitHub (pinned to 02eac45a1b)

Solutions

  1. Check the wrapped cause to distinguish network failure from executor-side kill failure.
  2. Confirm the worker is alive; if it crashed, the task instance can be marked as killed/failed manually via failover.
  3. Retry the kill; if the process is orphaned on the host, clean it up manually on the worker machine.
  4. Verify worker-group addresses and network paths between master and worker.

Example fix

// before
try { client.kill(taskExecution); } catch (Exception e) { /* swallowed */ }

// after
try {
    client.kill(taskExecution);
} catch (TaskKillException e) {
    log.error("Kill of task {} failed, cause: {}", taskExecution.getName(), e.getCause().getMessage());
    taskExecution.getTaskInstance().setState(TaskExecutionStatus.FAILURE);
}
Defensive patterns

Strategy: try-catch

Validate before calling

// confirm the task host is still registered
boolean workerAlive = registryClient.checkNodeExists(taskExecution.getHost(), NodeType.WORKER);
if (!workerAlive) {
    log.warn("Worker down; mark task {} failed instead of killing", taskExecution.getName());
}

Try / catch

try {
    client.kill(taskExecution);
} catch (TaskKillException e) {
    log.error("Kill of {} failed, cause: {}", taskExecution.getName(), e.getCause().getMessage());
    // mark task as failed or orphan-cleanup on the worker host
}

Prevention

When it happens

Trigger: Calling kill(ITaskExecution) when the executor is unreachable, the RPC throws (timeout, connection refused), or the delegator errors while forwarding the kill.

Common situations: User kills a workflow while the worker hosting the task is down; task process is in uninterruptible state; master-worker network split; worker restarted and the task registry entry is stale.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/7185737d4fa28122. Report an issue: GitHub.