{"record":{"id":"b7681bd80d62a30e","repo":"xuxueli/xxl-job","slug":"xxl-job-embedserver-bizthreadpool-is-exhausted","errorCode":null,"errorMessage":"xxl-job, EmbedServer bizThreadPool is EXHAUSTED!","messagePattern":"xxl-job, EmbedServer bizThreadPool is EXHAUSTED!","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"xxl-job-core/src/main/java/com/xxl/job/core/server/EmbedServer.java","lineNumber":72,"sourceCode":"                // param\n                EventLoopGroup bossGroup = new NioEventLoopGroup();\n                EventLoopGroup workerGroup = new NioEventLoopGroup();\n                ThreadPoolExecutor bizThreadPool = new ThreadPoolExecutor(\n                        0,\n                        200,\n                        60L,\n                        TimeUnit.SECONDS,\n                        new LinkedBlockingQueue<Runnable>(2000),\n                        new ThreadFactory() {\n                            @Override\n                            public Thread newThread(Runnable r) {\n                                return new Thread(r, \"xxl-job, EmbedServer bizThreadPool-\" + r.hashCode());\n                            }\n                        },\n                        new RejectedExecutionHandler() {\n                            @Override\n                            public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {\n                                throw new RuntimeException(\"xxl-job, EmbedServer bizThreadPool is EXHAUSTED!\");\n                            }\n                        });\n                try {\n                    // start server\n                    ServerBootstrap bootstrap = new ServerBootstrap();\n                    bootstrap.group(bossGroup, workerGroup)\n                            .channel(NioServerSocketChannel.class)\n                            .childHandler(new ChannelInitializer<SocketChannel>() {\n                                @Override\n                                public void initChannel(SocketChannel channel) throws Exception {\n                                    channel.pipeline()\n                                            .addLast(new IdleStateHandler(0, 0, 30 * 3, TimeUnit.SECONDS))  // beat 3N, close if idle\n                                            .addLast(new HttpServerCodec())\n                                            .addLast(new HttpObjectAggregator(5 * 1024 * 1024))  // merge request & reponse to FULL\n                                            .addLast(new EmbedHttpServerHandler(executorBiz, xxlJobExecutor, bizThreadPool));\n                                }\n                            })\n                            .childOption(ChannelOption.SO_KEEPALIVE, true);","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/xuxueli/xxl-job/blob/e74c784f68f81fa89cb350913ef15794865d7b12/xxl-job-core/src/main/java/com/xxl/job/core/server/EmbedServer.java#L54-L90","documentation":"xxl-job's executor embeds a Netty HTTP server whose business pool (core=0, max=200, queue=LinkedBlockingQueue(2000)) is shared by every inbound admin request — /trigger, /beat, /idleBeat, /kill, /log. When all 200 threads are busy AND the 2000-slot queue is full, ThreadPoolExecutor hands the task to the custom RejectedExecutionHandler at EmbedServer.java:69-74, which throws a bare RuntimeException(\"...bizThreadPool is EXHAUSTED!\"). Because execute() is called from EmbedHttpServerHandler.channelRead0 (line 169) on a Netty worker thread, the throw is not visible to your code: it propagates into Netty's exceptionCaught (lines 251-253), gets logged, and the channel is closed — so the admin side simply sees a dropped/failed request, not this message.","triggerScenarios":"The admin dispatches a /trigger (or /beat, /idleBeat, /kill, /log) to this executor while (200 threads executing) + (2000 queued tasks) is already reached. Concretely: a large broadcast/sharding trigger firing many sub-tasks at once; many jobs running minutes each so threads never recycle; a /log log-pull storm while jobs run; admin retry/traffic converging on one executor; recursive triggers or jobs that re-call the same executor's endpoints.","commonSituations":"(1) Broadcast/sharding job whose shards are long-running and fire together. (2) Routing strategy misconfigured so traffic funnels to one instance (e.g. only one executor registered under the appname, or FIRST/ROUND not spread). (3) Job handlers do unbounded blocking work (DB locks, downstream HTTP with no timeout), pinning all 200 threads. (4) Slow log storage causing /log pulls to pile up alongside triggers. (5) Version change from older xxl-job where pool/queue sizes differed. (6) Executor host CPU/thread-starved by a co-located app.","solutions":["Identify which endpoint floods: grep executor logs for the line, then correlate with admin trigger/beat/log volume. /beat storms usually mean admin is retrying against an executor it thinks is offline.","Scale out executors: register more instances under the same appname and set routeStrategy to ROUND or SHARDING_BROADCAST so no single executor absorbs all /trigger traffic.","Fix slow job handlers: cap downstream calls (HTTP/DB) with timeouts, release DB locks fast, avoid Thread.sleep/long loops in handlers so the 200 threads recycle.","Throttle at the source: lower admin schedule rate / blockStrategy, reduce concurrent batch sizes, add backoff so the executor drains its queue.","If you fork xxl-job-core: raise maximumPoolSize and/or queue capacity in EmbedServer (lines 57-62) only with headroom in heap and CPU; otherwise scale instances instead.","Monitor pool saturation (activeCount, queue.size) and alert before rejection; add a circuit breaker on the admin side to back off when an executor stops responding."],"exampleFix":"// before: handler holds a bizThreadPool thread for minutes\npublic ReturnT<String> execute() {\n    return longBlockingHttpCall();   // no timeout -> pins a worker\n}\n\n// after: cap downstream time so the thread returns to the pool\npublic ReturnT<String> execute() {\n    return withTimeout(longBlockingHttpCall(), Duration.ofSeconds(30));\n}\n\n// --- scaling fix (config, not code) ---\n// before: one executor instance carries all triggers\n// after: N instances under same appname, admin routeStrategy = ROUND","handlingStrategy":"retry","validationCode":"// xxl-job exposes no pre-call pool probe; you can only observe the symptom\n// (admin side, before each /trigger batch):\nExecutorBizClient client = ...; // admin->executor proxy\nResponse<String> beat = client.beat();\nif (beat == null || beat.getCode() != Response.SUCCESS_CODE) {\n    // executor unreachable / overloaded — back off instead of firing triggers\n    return; // or schedule retry with exponential backoff\n}","typeGuard":"// n/a — Java; rejection is a runtime condition, not a type-narrowing concern","tryCatchPattern":"// IMPORTANT: you CANNOT catch this RuntimeException at the call site —\n// it is thrown on the executor's Netty worker thread, swallowed by\n// EmbedHttpServerHandler.exceptionCaught (EmbedServer.java:251-253), and\n// surfaces to the caller only as a failed/closed connection. Handle the symptom:\ntry {\n    Response<String> r = executorClient.trigger(triggerParam);  // admin -> executor\n    if (r == null || r.getCode() != Response.SUCCESS_CODE) {\n        backoffAndRetry(triggerParam);  // exponential backoff + jitter\n    }\n} catch (Exception connFailure) {     // connect reset / read timeout\n    backoffAndRetry(triggerParam);\n}","preventionTips":["Run multiple executor instances behind one appname and use ROUND/SHARDING_BROADCAST routing so load is spread, never single-instance.","Keep job handlers short: every blocking call (HTTP, DB, lock) needs a timeout; never infinite-loop or Thread.sleep in execute().","Watch activeCount and queue size of the embedded pool (expose via JMX/metrics) and alert before the 200+2000 ceiling is hit.","Throttle the admin side: cap concurrent batch/broadcast size and schedule rate; back off when /beat starts failing.","Keep the executor host offloaded — don't co-locate CPU- or thread-hungry services that starve the Netty/pool threads.","On version upgrades, re-check EmbedServer pool/queue sizes; the ceiling is the contract you are tuning against."],"tags":["xxl-job","thread-pool","netty","concurrency","resource-exhaustion","java","scheduler","executor"],"backgroundTag":null,"analyzedSha":"e74c784f68f81fa89cb350913ef15794865d7b12","analyzedAt":"2026-08-14T04:22:43.715Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}