apache/dubbo · error · RejectedExecutionException

Thread pool is EXHAUSTED! Thread Name: %s, Pool Size: %d (ac

Error message

Thread pool is EXHAUSTED! Thread Name: %s, Pool Size: %d (active: %d, core: %d, max: %d, largest: %d), Task: %d (completed: %d), Executor status:(isShutdown:%s, isTerminated:%s, isTerminating:%s), in %s://%s:%d!

What it means

Thrown as RejectedExecutionException by AbortPolicyWithReport.rejectedExecution() when a Dubbo thread pool cannot accept a new task — all threads are busy and the work queue is full. The message includes detailed pool diagnostics (active/core/max/largest sizes, task counts, executor shutdown status, and the service URL). This is the default rejection handler for fixed/cached/bounded Dubbo thread pools. The logger also triggers a jstack dump (throttled to every 10 minutes) if dump.enable is true (default).

Source

Thrown at dubbo-common/src/main/java/org/apache/dubbo/common/threadpool/support/AbortPolicyWithReport.java:131

                e.getTaskCount(),
                e.getCompletedTaskCount(),
                e.isShutdown(),
                e.isTerminated(),
                e.isTerminating(),
                url.getProtocol(),
                url.getIp(),
                url.getPort());

        // 0-1 - Thread pool is EXHAUSTED!
        logger.warn(COMMON_THREAD_POOL_EXHAUSTED, "too much client requesting provider", "", msg);

        if (Boolean.parseBoolean(url.getParameter(DUMP_ENABLE, Boolean.TRUE.toString()))) {
            dumpJStack();
        }

        dispatchThreadPoolExhaustedEvent(msg);

        throw new RejectedExecutionException(msg);
    }

    public void addThreadPoolExhaustedEventListener(ThreadPoolExhaustedListener listener) {
        listeners.add(listener);
    }

    public void removeThreadPoolExhaustedEventListener(ThreadPoolExhaustedListener listener) {
        listeners.remove(listener);
    }

    /**
     * dispatch ThreadPoolExhaustedEvent
     *
     * @param msg
     */
    public void dispatchThreadPoolExhaustedEvent(String msg) {
        listeners.forEach(listener -> listener.onEvent(new ThreadPoolExhaustedEvent(msg)));
    }

View on GitHub (pinned to 3a3043227f)

Solutions

1. Increase thread pool size: set threads (threadpool.threads) and/or use a larger threadpool — e.g., dubbo.provider.threadpool.threads=200, or switch threadpool type to 'cached' for many short tasks.
2. Identify and eliminate slow/blocking operations in the Dubbo invocation path — move long-running logic to a separate executor, use Dubbo async invocation (CompletableFuture), or add timeouts.
3. Check the jstack dump file (in user.home or configured dump.directory) that Dubbo auto-generates — it shows exactly what every thread is doing at the point of exhaustion.
4. Scale horizontally by adding more provider instances, or enable connection/threads load balancing across endpoints.
5. Set appropriate timeouts (timeout parameter) so blocked invocations don't hold threads indefinitely.

Example fix

// before — default fixed thread pool with 200 threads
@dubbo.Service(threadpool = "fixed", threads = 200)
public class MyServiceImpl implements MyService { ... }

// after — larger pool + async to avoid blocking
@dubbo.Service(threadpool = "cached", threads = 500, timeout = 3000)
public class MyServiceImpl implements MyService {
    public CompletableFuture<Result> handle(Request req) {
        return CompletableFuture.supplyAsync(() -> doWork(req), separateExecutor);
    }
}

// Before submitting, check if the pool can accept work
public static boolean canSubmit(ThreadPoolExecutor pool) {
    return !pool.isShutdown()
        && pool.getActiveCount() < pool.getMaximumPoolSize()
        || pool.getQueue().remainingCapacity() > 0;
}

if (canSubmit(pool)) {
    pool.execute(task);
} else {
    // apply backpressure, queue externally, or reject gracefully
    handleOverload(task);
}

try {
    pool.execute(task);
} catch (RejectedExecutionException e) {
    if (e.getMessage().contains("EXHAUSTED")) {
        logger.warn("Thread pool exhausted, applying backpressure", e);
        // back off, queue externally, or drop with metrics
        overloadMeter.mark();
        handleOverload(task); // e.g., return error to caller, persist for retry
    } else {
        throw e;
    }
}

When it happens

Trigger: A provider or consumer thread pool is saturated: all threads are occupied by long-running or blocked tasks, and the queue is full, so the AbortPolicy rejects the next submitted task. Specifically when ThreadPoolExecutor.rejects via AbortPolicy (queue full + threads at max). Occurs on the provider side under burst load or on the consumer side when callback/async threads are exhausted.

Common situations: Slow downstream calls (database, external API) holding provider threads; burst traffic exceeding threadpool capacity; undersized thread pool configuration (threads, threadspool); business logic with blocking I/O on Dubbo threads instead of using async/NIO; deadlocks or thread starvation in the application; thread pool sized too small relative to request throughput.

Related errors

• Queue capacity is full.
• Executor is shutdown!
• Failed to acquire read lock on invokerRefreshLock within tim
• no more memory can be used !
• The task queue does not have executor!

AI-assisted analysis of apache/dubbo@3a3043227f (2026-08-14). Data as JSON: /api/errors/1e80afc7ba4d11ee. Report an issue: GitHub.