pinpoint-apm/pinpoint · error

UNAVAILABLE

UNAVAILABLE

Error message

In-flight byte budget exhausted

What it means

GrpcOtlpTraceService.export performs global byte-based admission: before queuing an OTLP trace export it calls admissionBytes.tryAcquire(request.getSerializedSize()) against a bounded in-flight byte budget. When the budget is exhausted the request is rejected with Status.UNAVAILABLE (retryable, so the OTLP exporter backs off), a warning is logged with the request size and budget, an INFLIGHT_BYTES rejection is recorded in metrics, and safeOnError returns ADMISSION_REJECTED to the client. This bounds collector memory regardless of request size or connection count.

Source

Thrown at otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/GrpcOtlpTraceService.java:74

        this.exportService = Objects.requireNonNull(exportService, "exportService");
        this.workerExecutor = Objects.requireNonNull(workerExecutor, "workerExecutor");
        this.ingestMetrics = Objects.requireNonNull(ingestMetrics, "ingestMetrics");
        this.maxInFlightBytes = maxInFlightBytes;
        this.admissionBytes = new Semaphore(maxInFlightBytes);
        // reserved = budget - free permits; sampled per export step (see OtlpTraceIngestMetrics).
        ingestMetrics.registerInFlightBytes(OtlpTraceIngestMetrics.Transport.GRPC,
                () -> (long) maxInFlightBytes - admissionBytes.availablePermits(), maxInFlightBytes);
    }

    @Override
    public void export(ExportTraceServiceRequest request, StreamObserver<ExportTraceServiceResponse> responseObserver) {
        // Global byte-based admission: reserve this request's wire size from the in-flight budget
        // before queuing. When the budget is exhausted, reject with UNAVAILABLE (retryable) so the
        // exporter backs off, bounding total memory regardless of request size / connection count.
        // (request size is already <= maxInboundMessageSize, enforced by gRPC before this handler.)
        final int requestBytes = request.getSerializedSize();
        if (!admissionBytes.tryAcquire(requestBytes)) {
            logger.warn("Failed to export. In-flight byte budget exhausted. requestBytes={}, budget={}", requestBytes, maxInFlightBytes);
            ingestMetrics.requestRejected(OtlpTraceIngestMetrics.Transport.GRPC, OtlpTraceIngestMetrics.RequestRejectReason.INFLIGHT_BYTES);
            safeOnError(responseObserver, ADMISSION_REJECTED);
            return;
        }

        ingestMetrics.requestBytes(OtlpTraceIngestMetrics.Transport.GRPC, requestBytes);

        final List<ResourceSpans> resourceSpanList = request.getResourceSpansList();
        // Offload the mapping/insert work onto the worker pool so the gRPC handler thread
        // (server.executor) is not blocked. A saturated worker queue surfaces to the client
        // as UNAVAILABLE (retryable), providing backpressure instead of dropping spans.
        final Context current = Context.current();
        final Runnable exportTask = current.wrap(() -> {
            try {
                if (Context.current().isCancelled()) {
                    // Client already gave up (deadline exceeded / cancelled) before this task ran.
                    // Skip the wasted mapping/insert; the admission reservation is freed in finally.
                    return;

View on GitHub (pinned to 744c3d3075)

Solutions

1. Treat UNAVAILABLE as retryable: configure the OTLP exporter with exponential backoff and honor retry hints.
2. Reduce per-export payload size (smaller BatchSpanProcessor max_export_batch_size / max_queue_size).
3. Increase the collector's maxInFlightBytes admission budget or add collector replicas to spread load.
4. Inspect ingestMetrics INFLIGHT_BYTES rejection counts to size the budget against real peak traffic.

Example fix

// before (client)
OtlpGrpcSpanExporter.builder().build(); // default retry, immediate

// after
OtlpGrpcSpanExporter.builder()
    .setTimeout(Duration.ofSeconds(10))
    .build(); // SDK retries UNAVAILABLE with exponential backoff

// gRPC UNAVAILABLE is the signal; enable SDK retry/backoff
onError(throwable -> {
    if (Status.fromThrowable(throwable).getCode() == Status.UNAVAILABLE.getCode()) {
        scheduleRetryWithExponentialBackoff();
    }
});

When it happens

Trigger: A gRPC Export call arrives while the sum of already-admitted in-flight serialized request bytes is within maxInflightBytes of the cap, so tryAcquire(requestBytes) fails. Requests larger than the remaining budget are rejected even if smaller ones would fit.

Common situations: Traffic spikes from many SDK instances, one very large batch consuming most of the budget, collector downscaling reducing capacity, or a client that ignores UNAVAILABLE and retries without backoff, keeping the budget saturated.

Related errors

• Failed to export. Worker executor rejected.
• EXECUTOR_REJECTED
• list size not same
• spanEventList is empty.
• first SpanEvent is null

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/5de530f6d222465e. Report an issue: GitHub.