xuxueli/xxl-job · error · RuntimeException

xxl-job, EmbedServer bizThreadPool is EXHAUSTED!

Error message

xxl-job, EmbedServer bizThreadPool is EXHAUSTED!

What it means

xxl-job's executor embeds a Netty HTTP server whose business pool (core=0, max=200, queue=LinkedBlockingQueue(2000)) is shared by every inbound admin request — /trigger, /beat, /idleBeat, /kill, /log. When all 200 threads are busy AND the 2000-slot queue is full, ThreadPoolExecutor hands the task to the custom RejectedExecutionHandler at EmbedServer.java:69-74, which throws a bare RuntimeException("...bizThreadPool is EXHAUSTED!"). Because execute() is called from EmbedHttpServerHandler.channelRead0 (line 169) on a Netty worker thread, the throw is not visible to your code: it propagates into Netty's exceptionCaught (lines 251-253), gets logged, and the channel is closed — so the admin side simply sees a dropped/failed request, not this message.

Source

Thrown at xxl-job-core/src/main/java/com/xxl/job/core/server/EmbedServer.java:72

                // param
                EventLoopGroup bossGroup = new NioEventLoopGroup();
                EventLoopGroup workerGroup = new NioEventLoopGroup();
                ThreadPoolExecutor bizThreadPool = new ThreadPoolExecutor(
                        0,
                        200,
                        60L,
                        TimeUnit.SECONDS,
                        new LinkedBlockingQueue<Runnable>(2000),
                        new ThreadFactory() {
                            @Override
                            public Thread newThread(Runnable r) {
                                return new Thread(r, "xxl-job, EmbedServer bizThreadPool-" + r.hashCode());
                            }
                        },
                        new RejectedExecutionHandler() {
                            @Override
                            public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
                                throw new RuntimeException("xxl-job, EmbedServer bizThreadPool is EXHAUSTED!");
                            }
                        });
                try {
                    // start server
                    ServerBootstrap bootstrap = new ServerBootstrap();
                    bootstrap.group(bossGroup, workerGroup)
                            .channel(NioServerSocketChannel.class)
                            .childHandler(new ChannelInitializer<SocketChannel>() {
                                @Override
                                public void initChannel(SocketChannel channel) throws Exception {
                                    channel.pipeline()
                                            .addLast(new IdleStateHandler(0, 0, 30 * 3, TimeUnit.SECONDS))  // beat 3N, close if idle
                                            .addLast(new HttpServerCodec())
                                            .addLast(new HttpObjectAggregator(5 * 1024 * 1024))  // merge request & reponse to FULL
                                            .addLast(new EmbedHttpServerHandler(executorBiz, xxlJobExecutor, bizThreadPool));
                                }
                            })
                            .childOption(ChannelOption.SO_KEEPALIVE, true);

View on GitHub (pinned to e74c784f68)

Solutions

1. Identify which endpoint floods: grep executor logs for the line, then correlate with admin trigger/beat/log volume. /beat storms usually mean admin is retrying against an executor it thinks is offline.
2. Scale out executors: register more instances under the same appname and set routeStrategy to ROUND or SHARDING_BROADCAST so no single executor absorbs all /trigger traffic.
3. Fix slow job handlers: cap downstream calls (HTTP/DB) with timeouts, release DB locks fast, avoid Thread.sleep/long loops in handlers so the 200 threads recycle.
4. Throttle at the source: lower admin schedule rate / blockStrategy, reduce concurrent batch sizes, add backoff so the executor drains its queue.
5. If you fork xxl-job-core: raise maximumPoolSize and/or queue capacity in EmbedServer (lines 57-62) only with headroom in heap and CPU; otherwise scale instances instead.
6. Monitor pool saturation (activeCount, queue.size) and alert before rejection; add a circuit breaker on the admin side to back off when an executor stops responding.

Example fix

// before: handler holds a bizThreadPool thread for minutes
public ReturnT<String> execute() {
    return longBlockingHttpCall();   // no timeout -> pins a worker
}

// after: cap downstream time so the thread returns to the pool
public ReturnT<String> execute() {
    return withTimeout(longBlockingHttpCall(), Duration.ofSeconds(30));
}

// --- scaling fix (config, not code) ---
// before: one executor instance carries all triggers
// after: N instances under same appname, admin routeStrategy = ROUND

// xxl-job exposes no pre-call pool probe; you can only observe the symptom
// (admin side, before each /trigger batch):
ExecutorBizClient client = ...; // admin->executor proxy
Response<String> beat = client.beat();
if (beat == null || beat.getCode() != Response.SUCCESS_CODE) {
    // executor unreachable / overloaded — back off instead of firing triggers
    return; // or schedule retry with exponential backoff
}

// n/a — Java; rejection is a runtime condition, not a type-narrowing concern

// IMPORTANT: you CANNOT catch this RuntimeException at the call site —
// it is thrown on the executor's Netty worker thread, swallowed by
// EmbedHttpServerHandler.exceptionCaught (EmbedServer.java:251-253), and
// surfaces to the caller only as a failed/closed connection. Handle the symptom:
try {
    Response<String> r = executorClient.trigger(triggerParam);  // admin -> executor
    if (r == null || r.getCode() != Response.SUCCESS_CODE) {
        backoffAndRetry(triggerParam);  // exponential backoff + jitter
    }
} catch (Exception connFailure) {     // connect reset / read timeout
    backoffAndRetry(triggerParam);
}

When it happens

Trigger: The admin dispatches a /trigger (or /beat, /idleBeat, /kill, /log) to this executor while (200 threads executing) + (2000 queued tasks) is already reached. Concretely: a large broadcast/sharding trigger firing many sub-tasks at once; many jobs running minutes each so threads never recycle; a /log log-pull storm while jobs run; admin retry/traffic converging on one executor; recursive triggers or jobs that re-call the same executor's endpoints.

Common situations: (1) Broadcast/sharding job whose shards are long-running and fire together. (2) Routing strategy misconfigured so traffic funnels to one instance (e.g. only one executor registered under the appname, or FIRST/ROUND not spread). (3) Job handlers do unbounded blocking work (DB locks, downstream HTTP with no timeout), pinning all 200 threads. (4) Slow log storage causing /log pulls to pile up alongside triggers. (5) Version change from older xxl-job where pool/queue sizes differed. (6) Executor host CPU/thread-starved by a co-located app.

Related errors

• xxl-job executor adminAddresses empty.
• xxl-job executor appname empty.
• xxl-job executor accessToken empty.

AI-assisted analysis of xuxueli/xxl-job@e74c784f68 (2026-08-14). Data as JSON: /api/errors/b7681bd80d62a30e. Report an issue: GitHub.