pinpoint-apm/pinpoint · warning

EXECUTOR_REJECTED

EXECUTOR_REJECTED

Error message

Failed to request. Executor rejected. header:{}

What it means

SpanService.sendSpanBatch submits span-message batch handling as a task to the collector's executor. When the executor's queue/thread limits are exhausted, executor.execute throws RejectedExecutionException; the service logs this warning with the client agent info and responds with an EXECUTOR_REJECTED error on the gRPC stream, telling the agent its span batch was not accepted.

Source

Thrown at collector/src/main/java/com/navercorp/pinpoint/collector/receiver/grpc/service/SpanService.java:89

                       ServerRequestFactory serverRequestFactory,
                       StreamCloseOnError streamCloseOnError) {
        this.spanHandler = Objects.requireNonNull(spanHandler, "spanHandler");
        this.spanCheckHandler = Objects.requireNonNull(spanCheckHandler, "spanCheckHandler");

        this.uidFetcherStreamService = Objects.requireNonNull(uidFetcherStreamService, "uidFetcherStreamService");
        this.executor = Objects.requireNonNull(executor, "executor");
        this.serverRequestFactory = Objects.requireNonNull(serverRequestFactory, "serverRequestFactory");
        this.streamCloseOnError = Objects.requireNonNull(streamCloseOnError, "streamCloseOnError");
    }

    @Override
    public void sendSpanBatch(PSpanMessageBatch request, StreamObserver<PSpanResultBatch> responseObserver) {
        final Context current = Context.current();
        final Runnable batchTask = current.wrap(() -> handleSpanBatch(current, request, responseObserver));
        try {
            executor.execute(batchTask);
        } catch (RejectedExecutionException e) {
            logger.warn("Failed to request. Executor rejected. header:{}", ServerContext.getAgentInfo(current));
            responseObserver.onError(EXECUTOR_REJECTED.asException());
        }
    }

    private void handleSpanBatch(Context current, PSpanMessageBatch request, StreamObserver<PSpanResultBatch> responseObserver) {
        final UidFetcher fetcher = uidFetcherStreamService.newUidFetcher();
        final SpanBatchErrorResult errorReporter = new SpanBatchErrorResult();
        final String serviceName = ServerContext.getAgentInfo(current).getServiceName();
        if (serviceNotFoundChecker.isServiceNotFoundNow(serviceName, fetcher)) {
            // discard silently
            responseObserver.onNext(PSpanResultBatch.getDefaultInstance());
            responseObserver.onCompleted();
            return;
        }
        for (PSpanMessage spanMessage : request.getSpanList()) {
            if (isDebug) {
                logger.debug("SendSpanList PSpanMessage={}", MessageFormatUtils.debugLog(spanMessage));
            }

View on GitHub (pinned to 744c3d3075)

Solutions

1. Increase the span executor's worker thread count and queue size in collector configuration to absorb bursts
2. Scale out collector instances and rebalance agent traffic across them
3. Check upstream/downstream latency (e.g., to the storage layer) that keeps executor workers busy; fix slow consumers
4. Enable or verify agent-side retry/backoff so rejected batches are resent rather than lost
5. If the error appears during shutdown, drain/stop traffic before stopping collectors

Example fix

// before (collector config)
collector.span.executors=4
collector.span.executors.queue=1024
// after
collector.span.executors=16
collector.span.executors.queue=8192

// Client side: back off when the collector signals EXECUTOR_REJECTED
if (status.getCode() == Status.Code.RESOURCE_EXHAUSTED && "EXECUTOR_REJECTED".equals(status.getDescription())) {
    scheduleRetryWithBackoff(batch);
}

try {
    spanStub.sendSpanMessageBatch(request);
} catch (StatusRuntimeException e) {
    if (e.getStatus().getCode() == Status.Code.RESOURCE_EXHAUSTED) {
        retryWithExponentialBackoff(request, maxAttempts);
    } else {
        throw e;
    }
}

When it happens

Trigger: A gRPC sendSpanBatch call arrives while the collector's span executor is saturated (all workers busy and the work queue at capacity, or the executor is shut down), causing RejectedExecutionException from executor.execute(batchTask).

Common situations: Span traffic spike/agent fleet growth overwhelming collector worker queue sizes (executor.time / queue capacity settings); collector restart or shutdown rejecting in-flight RPCs; downstream slowness (e.g., storage writes) backing up the executor; misconfigured thread pool too small for the load.

Related errors

• Failed to export. Worker executor rejected.
• INTERNAL_SERVER_ERROR
• INTERNAL_SERVER_ERROR
• INTERNAL_SERVER_ERROR
• INTERNAL_SERVER_ERROR

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/8fd81a288e12bb64. Report an issue: GitHub.