apache/dolphinscheduler · error · TaskExecutorRuntimeException

All ExclusiveThreadTaskExecutorWorker are busy

Error message

All ExclusiveThreadTaskExecutorWorker are busy

What it means

AbstractTaskExecutorContainer.dispatch assigns each task executor to an exclusive per-thread worker. When getTaskExecutorWorkerCandidate finds no free worker (all worker threads are occupied by other task executors), dispatch throws TaskExecutorRuntimeException instead of queueing, providing backpressure to the submitter.

Solutions

  1. Increase the container's worker thread count / parallelism configuration
  2. Reduce concurrent workflow/task submission or add queueing before dispatch
  3. Retry dispatch after a delay (or monitor worker availability before submitting)
  4. Scale out masters to spread the load

Example fix

// before
taskExecutorContainer.dispatch(taskExecutor); // may throw when busy
// after
try {
    taskExecutorContainer.dispatch(taskExecutor);
} catch (TaskExecutorRuntimeException e) {
    // requeue and retry later
    pendingExecutors.offer(taskExecutor);
}
Defensive patterns

Strategy: fallback

Validate before calling

// Throttle submissions to the container's worker count:
if (activeDispatched >= configuredWorkerThreads) {
    pendingExecutors.offer(taskExecutor); // queue instead of dispatching now
    return;
}

Try / catch

try {
    container.dispatch(taskExecutor);
} catch (TaskExecutorRuntimeException e) {
    if (e.getMessage().contains("are busy")) {
        pendingExecutors.offer(taskExecutor); // retry with backoff
    } else throw e;
}

Prevention

When it happens

Trigger: Submitting more concurrent task executors to the container than there are worker threads: dispatch() called while every ExclusiveThreadTaskExecutorWorker already has a registered executor.

Common situations: Burst of task instances exceeding workerThreadNum/parallelism configured for the container; long-running tasks hogging all workers; under-provisioned master with high workflow concurrency.

Related errors

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/29a52181ccbb278d. Report an issue: GitHub.

Appendix: source

Thrown at dolphinscheduler-task-executor/src/main/java/org/apache/dolphinscheduler/task/executor/container/AbstractTaskExecutorContainer.java:63

    protected final TaskExecutorWorkers taskExecutorWorkers;

    public AbstractTaskExecutorContainer(final TaskExecutorContainerConfig containerConfig) {
        final String threadPoolFormat = containerConfig.getContainerName() + "-worker-%d";
        final int threadPoolSize = containerConfig.getTaskExecutorThreadPoolSize();
        this.taskExecutorThreadPool = ThreadUtils.newDaemonFixedThreadExecutor(threadPoolFormat, threadPoolSize);
        this.taskExecutorWorkers = new TaskExecutorWorkers(threadPoolSize);
        this.taskExecutorAssignmentTable = new TaskExecutorAssignmentTable();
        startAllThreadTaskExecutorWorker();
    }

    @Override
    public void dispatch(final ITaskExecutor taskExecutor) {
        synchronized (this) {
            Optional<TaskExecutorWorker> taskExecutorWorkerCandidate = getTaskExecutorWorkerCandidate(taskExecutor);
            if (!taskExecutorWorkerCandidate.isPresent()) {
                log.info("All ExclusiveThreadTaskExecutorWorker are busy, cannot submit taskExecutor(id={})",
                        taskExecutor.getId());
                throw new TaskExecutorRuntimeException("All ExclusiveThreadTaskExecutorWorker are busy");
            }
            final TaskExecutorWorker taskExecutorWorker = taskExecutorWorkerCandidate.get();
            taskExecutorWorker.registerTaskExecutor(taskExecutor);
            taskExecutorAssignmentTable.registerTaskExecutor(taskExecutor, taskExecutorWorker);
        }
    }

    @Override
    public void start(final ITaskExecutor taskExecutor) {
        final Integer workerId = taskExecutorAssignmentTable.getTaskExecutorWorkerId(taskExecutor);
        if (workerId == null) {
            throw new IllegalStateException(
                    "The taskExecutor: " + taskExecutor.getId() + " is not registered to any worker");
        }
        final TaskExecutorWorker taskExecutorWorker = taskExecutorWorkers.getWorkerById(workerId);
        taskExecutorWorker.fireTaskExecutor(taskExecutor);
    }

View on GitHub (pinned to 02eac45a1b)