pinpoint-apm/pinpoint · error
Failed to export. Worker executor rejected.
Error message
Failed to export. Worker executor rejected.
What it means
GrpcOtlpTraceService.export submits the actual export work (exportTask) to a bounded workerExecutor. When the executor's queue/threads are saturated it throws RejectedExecutionException; the service then releases the previously acquired admission bytes, logs 'Failed to export. Worker executor rejected.', records an EXECUTOR_REJECTED request rejection metric, and completes the response observer with an UNAVAILABLE error (EXECUTOR_REJECTED) so the client can retry later. This is backpressure, not data corruption.
Source
Thrown at otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/GrpcOtlpTraceService.java:110
return;
}
handleExport(resourceSpanList, responseObserver);
} catch (Throwable t) {
// Unexpected failure (e.g. mapping fault on malformed input). Without this catch the
// exception would escape to the worker thread: the response would never be closed
// (client hangs until deadline) and the worker thread would die on an uncaught error.
// INTERNAL is non-retryable, avoiding a retry storm on deterministic (poison-data) faults.
logger.warn("Unexpected error while exporting otlp trace", t);
safeOnError(responseObserver, Status.INTERNAL.withDescription("export failed"));
} finally {
admissionBytes.release(requestBytes);
}
});
try {
workerExecutor.execute(exportTask);
} catch (RejectedExecutionException e) {
admissionBytes.release(requestBytes);
logger.warn("Failed to export. Worker executor rejected.");
ingestMetrics.requestRejected(OtlpTraceIngestMetrics.Transport.GRPC, OtlpTraceIngestMetrics.RequestRejectReason.EXECUTOR_REJECTED);
safeOnError(responseObserver, EXECUTOR_REJECTED);
}
}
private void handleExport(List<ResourceSpans> resourceSpanList, StreamObserver<ExportTraceServiceResponse> responseObserver) {
final OtlpTraceExportResult result = exportService.export(resourceSpanList, OtlpTraceIngestMetrics.Transport.GRPC);
if (OtlpTraceResponseMapper.isServerError(result)) {
// Server-side / transient failures (HBase insert, agentInfo): ask the client to retry
// the whole batch via UNAVAILABLE (retryable) instead of dropping recoverable data.
// INVALID_ARGUMENT here would be treated as non-retryable and silently lost.
safeOnError(responseObserver, Status.UNAVAILABLE.withDescription(result.serverMessage()));
return;
}
// Client-side data faults surface as OTLP partial success; a clean run as the empty response.
safeComplete(responseObserver, OtlpTraceResponseMapper.toResponse(result));View on GitHub (pinned to 744c3d3075)
Solutions
- Retry with backoff on the client side (UNAVAILABLE is retryable); the admission bytes were released so a later attempt can succeed.
- Increase workerExecutor thread count / queue capacity to match peak concurrency.
- Investigate why workers are slow (downstream sink latency) and fix that bottleneck first.
- Reduce client export concurrency/batch size, or scale out collector instances.
Example fix
// before
ExecutorService workerExecutor = Executors.newFixedThreadPool(4);
// after
ExecutorService workerExecutor = new ThreadPoolExecutor(
8, 8, 60L, TimeUnit.SECONDS,
new ArrayBlockingQueue<>(1024),
new ThreadPoolExecutor.AbortPolicy()); Defensive patterns
Strategy: retry
Try / catch
// client: treat UNAVAILABLE/EXECUTOR_REJECTED as transient RetryConfiguration: maxAttempts with exponential backoff + jitter on Status.Code.UNAVAILABLE; surface a dead-letter/log when attempts are exhausted.
Prevention
- Size the worker executor for peak concurrency, not average.
- Alert on collector executor saturation and downstream sink latency.
- Cap client-side concurrent export calls to a sane multiple of collector capacity.
- Fix slow downstream writes (the usual root cause of executor pile-up).
When it happens
Trigger: workerExecutor.execute(exportTask) throws RejectedExecutionException because the executor's queue is full and all worker threads are busy — i.e., more concurrent exports than the configured thread pool and queue capacity can absorb.
Common situations: Sudden traffic spikes, worker threads blocked on slow downstream writes (storage/Kafka), executor queue sized too small for the connection count, or a prolonged downstream outage causing tasks to pile up until rejection.
Understand the failure class
Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.
Related errors
AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07).
Data as JSON: /api/errors/6df30cb5d7cfc010.
Report an issue: GitHub.