{"record":{"id":"1e80afc7ba4d11ee","repo":"apache/dubbo","slug":"thread-pool-is-exhausted-thread-name-s-pool-si","errorCode":null,"errorMessage":"Thread pool is EXHAUSTED! Thread Name: %s, Pool Size: %d (active: %d, core: %d, max: %d, largest: %d), Task: %d (completed: %d), Executor status:(isShutdown:%s, isTerminated:%s, isTerminating:%s), in %s://%s:%d!","messagePattern":"Thread pool is EXHAUSTED! Thread Name: (.+?), Pool Size: (.+?) \\(active: (.+?), core: (.+?), max: (.+?), largest: (.+?)\\), Task: (.+?) \\(completed: (.+?)\\), Executor status:\\(isShutdown:(.+?), isTerminated:(.+?), isTerminating:(.+?)\\), in (.+?)://(.+?):(.+?)!","errorType":"exception","errorClass":"RejectedExecutionException","httpStatus":null,"severity":"error","filePath":"dubbo-common/src/main/java/org/apache/dubbo/common/threadpool/support/AbortPolicyWithReport.java","lineNumber":131,"sourceCode":"                e.getTaskCount(),\n                e.getCompletedTaskCount(),\n                e.isShutdown(),\n                e.isTerminated(),\n                e.isTerminating(),\n                url.getProtocol(),\n                url.getIp(),\n                url.getPort());\n\n        // 0-1 - Thread pool is EXHAUSTED!\n        logger.warn(COMMON_THREAD_POOL_EXHAUSTED, \"too much client requesting provider\", \"\", msg);\n\n        if (Boolean.parseBoolean(url.getParameter(DUMP_ENABLE, Boolean.TRUE.toString()))) {\n            dumpJStack();\n        }\n\n        dispatchThreadPoolExhaustedEvent(msg);\n\n        throw new RejectedExecutionException(msg);\n    }\n\n    public void addThreadPoolExhaustedEventListener(ThreadPoolExhaustedListener listener) {\n        listeners.add(listener);\n    }\n\n    public void removeThreadPoolExhaustedEventListener(ThreadPoolExhaustedListener listener) {\n        listeners.remove(listener);\n    }\n\n    /**\n     * dispatch ThreadPoolExhaustedEvent\n     *\n     * @param msg\n     */\n    public void dispatchThreadPoolExhaustedEvent(String msg) {\n        listeners.forEach(listener -> listener.onEvent(new ThreadPoolExhaustedEvent(msg)));\n    }","sourceCodeStart":113,"sourceCodeEnd":149,"githubUrl":"https://github.com/apache/dubbo/blob/3a3043227f5571d25eb2889de5bca22f2914843b/dubbo-common/src/main/java/org/apache/dubbo/common/threadpool/support/AbortPolicyWithReport.java#L113-L149","documentation":"Thrown as RejectedExecutionException by AbortPolicyWithReport.rejectedExecution() when a Dubbo thread pool cannot accept a new task — all threads are busy and the work queue is full. The message includes detailed pool diagnostics (active/core/max/largest sizes, task counts, executor shutdown status, and the service URL). This is the default rejection handler for fixed/cached/bounded Dubbo thread pools. The logger also triggers a jstack dump (throttled to every 10 minutes) if dump.enable is true (default).","triggerScenarios":"A provider or consumer thread pool is saturated: all threads are occupied by long-running or blocked tasks, and the queue is full, so the AbortPolicy rejects the next submitted task. Specifically when ThreadPoolExecutor.rejects via AbortPolicy (queue full + threads at max). Occurs on the provider side under burst load or on the consumer side when callback/async threads are exhausted.","commonSituations":"Slow downstream calls (database, external API) holding provider threads; burst traffic exceeding threadpool capacity; undersized thread pool configuration (threads, threadspool); business logic with blocking I/O on Dubbo threads instead of using async/NIO; deadlocks or thread starvation in the application; thread pool sized too small relative to request throughput.","solutions":["Increase thread pool size: set threads (threadpool.threads) and/or use a larger threadpool — e.g., dubbo.provider.threadpool.threads=200, or switch threadpool type to 'cached' for many short tasks.","Identify and eliminate slow/blocking operations in the Dubbo invocation path — move long-running logic to a separate executor, use Dubbo async invocation (CompletableFuture), or add timeouts.","Check the jstack dump file (in user.home or configured dump.directory) that Dubbo auto-generates — it shows exactly what every thread is doing at the point of exhaustion.","Scale horizontally by adding more provider instances, or enable connection/threads load balancing across endpoints.","Set appropriate timeouts (timeout parameter) so blocked invocations don't hold threads indefinitely."],"exampleFix":"// before — default fixed thread pool with 200 threads\n@dubbo.Service(threadpool = \"fixed\", threads = 200)\npublic class MyServiceImpl implements MyService { ... }\n\n// after — larger pool + async to avoid blocking\n@dubbo.Service(threadpool = \"cached\", threads = 500, timeout = 3000)\npublic class MyServiceImpl implements MyService {\n    public CompletableFuture<Result> handle(Request req) {\n        return CompletableFuture.supplyAsync(() -> doWork(req), separateExecutor);\n    }\n}","handlingStrategy":"try-catch","validationCode":"// Before submitting, check if the pool can accept work\npublic static boolean canSubmit(ThreadPoolExecutor pool) {\n    return !pool.isShutdown()\n        && pool.getActiveCount() < pool.getMaximumPoolSize()\n        || pool.getQueue().remainingCapacity() > 0;\n}\n\nif (canSubmit(pool)) {\n    pool.execute(task);\n} else {\n    // apply backpressure, queue externally, or reject gracefully\n    handleOverload(task);\n}","typeGuard":null,"tryCatchPattern":"try {\n    pool.execute(task);\n} catch (RejectedExecutionException e) {\n    if (e.getMessage().contains(\"EXHAUSTED\")) {\n        logger.warn(\"Thread pool exhausted, applying backpressure\", e);\n        // back off, queue externally, or drop with metrics\n        overloadMeter.mark();\n        handleOverload(task); // e.g., return error to caller, persist for retry\n    } else {\n        throw e;\n    }\n}","preventionTips":["Size thread pools based on measured peak load, not defaults.","Set invocation timeouts so blocked threads are released.","Monitor pool active count / queue size and alert before exhaustion.","Move blocking I/O off Dubbo threads using async (CompletableFuture) or dedicated executors.","Review the auto-generated jstack dump to identify what threads are doing when exhaustion occurs."],"tags":["thread-pool","performance","capacity","rejected-execution","provider"],"backgroundTag":null,"analyzedSha":"3a3043227f5571d25eb2889de5bca22f2914843b","analyzedAt":"2026-08-14T00:43:19.853Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}