{"record":{"id":"8fd81a288e12bb64","repo":"pinpoint-apm/pinpoint","slug":"executor-rejected","errorCode":"EXECUTOR_REJECTED","errorMessage":"Failed to request. Executor rejected. header:{}","messagePattern":"Failed to request\\. Executor rejected\\. header:(.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"collector/src/main/java/com/navercorp/pinpoint/collector/receiver/grpc/service/SpanService.java","lineNumber":89,"sourceCode":"                       ServerRequestFactory serverRequestFactory,\n                       StreamCloseOnError streamCloseOnError) {\n        this.spanHandler = Objects.requireNonNull(spanHandler, \"spanHandler\");\n        this.spanCheckHandler = Objects.requireNonNull(spanCheckHandler, \"spanCheckHandler\");\n\n        this.uidFetcherStreamService = Objects.requireNonNull(uidFetcherStreamService, \"uidFetcherStreamService\");\n        this.executor = Objects.requireNonNull(executor, \"executor\");\n        this.serverRequestFactory = Objects.requireNonNull(serverRequestFactory, \"serverRequestFactory\");\n        this.streamCloseOnError = Objects.requireNonNull(streamCloseOnError, \"streamCloseOnError\");\n    }\n\n    @Override\n    public void sendSpanBatch(PSpanMessageBatch request, StreamObserver<PSpanResultBatch> responseObserver) {\n        final Context current = Context.current();\n        final Runnable batchTask = current.wrap(() -> handleSpanBatch(current, request, responseObserver));\n        try {\n            executor.execute(batchTask);\n        } catch (RejectedExecutionException e) {\n            logger.warn(\"Failed to request. Executor rejected. header:{}\", ServerContext.getAgentInfo(current));\n            responseObserver.onError(EXECUTOR_REJECTED.asException());\n        }\n    }\n\n    private void handleSpanBatch(Context current, PSpanMessageBatch request, StreamObserver<PSpanResultBatch> responseObserver) {\n        final UidFetcher fetcher = uidFetcherStreamService.newUidFetcher();\n        final SpanBatchErrorResult errorReporter = new SpanBatchErrorResult();\n        final String serviceName = ServerContext.getAgentInfo(current).getServiceName();\n        if (serviceNotFoundChecker.isServiceNotFoundNow(serviceName, fetcher)) {\n            // discard silently\n            responseObserver.onNext(PSpanResultBatch.getDefaultInstance());\n            responseObserver.onCompleted();\n            return;\n        }\n        for (PSpanMessage spanMessage : request.getSpanList()) {\n            if (isDebug) {\n                logger.debug(\"SendSpanList PSpanMessage={}\", MessageFormatUtils.debugLog(spanMessage));\n            }","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/pinpoint-apm/pinpoint/blob/744c3d3075e595656abb1ae331ad2c0e4c9eb996/collector/src/main/java/com/navercorp/pinpoint/collector/receiver/grpc/service/SpanService.java#L71-L107","documentation":"SpanService.sendSpanBatch submits span-message batch handling as a task to the collector's executor. When the executor's queue/thread limits are exhausted, executor.execute throws RejectedExecutionException; the service logs this warning with the client agent info and responds with an EXECUTOR_REJECTED error on the gRPC stream, telling the agent its span batch was not accepted.","triggerScenarios":"A gRPC sendSpanBatch call arrives while the collector's span executor is saturated (all workers busy and the work queue at capacity, or the executor is shut down), causing RejectedExecutionException from executor.execute(batchTask).","commonSituations":"Span traffic spike/agent fleet growth overwhelming collector worker queue sizes (executor.time / queue capacity settings); collector restart or shutdown rejecting in-flight RPCs; downstream slowness (e.g., storage writes) backing up the executor; misconfigured thread pool too small for the load.","solutions":["Increase the span executor's worker thread count and queue size in collector configuration to absorb bursts","Scale out collector instances and rebalance agent traffic across them","Check upstream/downstream latency (e.g., to the storage layer) that keeps executor workers busy; fix slow consumers","Enable or verify agent-side retry/backoff so rejected batches are resent rather than lost","If the error appears during shutdown, drain/stop traffic before stopping collectors"],"exampleFix":"// before (collector config)\ncollector.span.executors=4\ncollector.span.executors.queue=1024\n// after\ncollector.span.executors=16\ncollector.span.executors.queue=8192","handlingStrategy":"retry","validationCode":"// Client side: back off when the collector signals EXECUTOR_REJECTED\nif (status.getCode() == Status.Code.RESOURCE_EXHAUSTED && \"EXECUTOR_REJECTED\".equals(status.getDescription())) {\n    scheduleRetryWithBackoff(batch);\n}","typeGuard":null,"tryCatchPattern":"try {\n    spanStub.sendSpanMessageBatch(request);\n} catch (StatusRuntimeException e) {\n    if (e.getStatus().getCode() == Status.Code.RESOURCE_EXHAUSTED) {\n        retryWithExponentialBackoff(request, maxAttempts);\n    } else {\n        throw e;\n    }\n}","preventionTips":["Size the collector span executor for peak span throughput with headroom","Keep agent-side retry with exponential backoff enabled for executor-rejected batches","Watch collector queue utilization metrics and scale before saturation","Fix slow span consumers (storage sinks) that keep the executor busy"],"tags":["grpc","overload","thread-pool","collector","backpressure"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"744c3d3075e595656abb1ae331ad2c0e4c9eb996","analyzedAt":"2026-09-07T18:48:45.289Z","contentChangedAt":"2026-09-07T18:48:45.289Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}