{"record":{"id":"6df30cb5d7cfc010","repo":"pinpoint-apm/pinpoint","slug":"failed-to-export-worker-executor-rejected","errorCode":null,"errorMessage":"Failed to export. Worker executor rejected.","messagePattern":"Failed to export\\. Worker executor rejected\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/GrpcOtlpTraceService.java","lineNumber":110,"sourceCode":"                    return;\n                }\n                handleExport(resourceSpanList, responseObserver);\n            } catch (Throwable t) {\n                // Unexpected failure (e.g. mapping fault on malformed input). Without this catch the\n                // exception would escape to the worker thread: the response would never be closed\n                // (client hangs until deadline) and the worker thread would die on an uncaught error.\n                // INTERNAL is non-retryable, avoiding a retry storm on deterministic (poison-data) faults.\n                logger.warn(\"Unexpected error while exporting otlp trace\", t);\n                safeOnError(responseObserver, Status.INTERNAL.withDescription(\"export failed\"));\n            } finally {\n                admissionBytes.release(requestBytes);\n            }\n        });\n        try {\n            workerExecutor.execute(exportTask);\n        } catch (RejectedExecutionException e) {\n            admissionBytes.release(requestBytes);\n            logger.warn(\"Failed to export. Worker executor rejected.\");\n            ingestMetrics.requestRejected(OtlpTraceIngestMetrics.Transport.GRPC, OtlpTraceIngestMetrics.RequestRejectReason.EXECUTOR_REJECTED);\n            safeOnError(responseObserver, EXECUTOR_REJECTED);\n        }\n    }\n\n    private void handleExport(List<ResourceSpans> resourceSpanList, StreamObserver<ExportTraceServiceResponse> responseObserver) {\n        final OtlpTraceExportResult result = exportService.export(resourceSpanList, OtlpTraceIngestMetrics.Transport.GRPC);\n\n        if (OtlpTraceResponseMapper.isServerError(result)) {\n            // Server-side / transient failures (HBase insert, agentInfo): ask the client to retry\n            // the whole batch via UNAVAILABLE (retryable) instead of dropping recoverable data.\n            // INVALID_ARGUMENT here would be treated as non-retryable and silently lost.\n            safeOnError(responseObserver, Status.UNAVAILABLE.withDescription(result.serverMessage()));\n            return;\n        }\n\n        // Client-side data faults surface as OTLP partial success; a clean run as the empty response.\n        safeComplete(responseObserver, OtlpTraceResponseMapper.toResponse(result));","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/pinpoint-apm/pinpoint/blob/744c3d3075e595656abb1ae331ad2c0e4c9eb996/otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/GrpcOtlpTraceService.java#L92-L128","documentation":"GrpcOtlpTraceService.export submits the actual export work (exportTask) to a bounded workerExecutor. When the executor's queue/threads are saturated it throws RejectedExecutionException; the service then releases the previously acquired admission bytes, logs 'Failed to export. Worker executor rejected.', records an EXECUTOR_REJECTED request rejection metric, and completes the response observer with an UNAVAILABLE error (EXECUTOR_REJECTED) so the client can retry later. This is backpressure, not data corruption.","triggerScenarios":"workerExecutor.execute(exportTask) throws RejectedExecutionException because the executor's queue is full and all worker threads are busy — i.e., more concurrent exports than the configured thread pool and queue capacity can absorb.","commonSituations":"Sudden traffic spikes, worker threads blocked on slow downstream writes (storage/Kafka), executor queue sized too small for the connection count, or a prolonged downstream outage causing tasks to pile up until rejection.","solutions":["Retry with backoff on the client side (UNAVAILABLE is retryable); the admission bytes were released so a later attempt can succeed.","Increase workerExecutor thread count / queue capacity to match peak concurrency.","Investigate why workers are slow (downstream sink latency) and fix that bottleneck first.","Reduce client export concurrency/batch size, or scale out collector instances."],"exampleFix":"// before\nExecutorService workerExecutor = Executors.newFixedThreadPool(4);\n\n// after\nExecutorService workerExecutor = new ThreadPoolExecutor(\n    8, 8, 60L, TimeUnit.SECONDS,\n    new ArrayBlockingQueue<>(1024),\n    new ThreadPoolExecutor.AbortPolicy());","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// client: treat UNAVAILABLE/EXECUTOR_REJECTED as transient\nRetryConfiguration: maxAttempts with exponential backoff + jitter on Status.Code.UNAVAILABLE;\nsurface a dead-letter/log when attempts are exhausted.","preventionTips":["Size the worker executor for peak concurrency, not average.","Alert on collector executor saturation and downstream sink latency.","Cap client-side concurrent export calls to a sane multiple of collector capacity.","Fix slow downstream writes (the usual root cause of executor pile-up)."],"tags":["grpc","otlp","thread-pool","backpressure","rejected-execution"],"backgroundTag":"request-timeout","analyzedSha":"744c3d3075e595656abb1ae331ad2c0e4c9eb996","analyzedAt":"2026-09-07T18:48:45.289Z","contentChangedAt":"2026-09-07T18:48:45.289Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}