{"record":{"id":"5de530f6d222465e","repo":"pinpoint-apm/pinpoint","slug":"unavailable","errorCode":"UNAVAILABLE","errorMessage":"In-flight byte budget exhausted","messagePattern":"In-flight byte budget exhausted","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/GrpcOtlpTraceService.java","lineNumber":74,"sourceCode":"        this.exportService = Objects.requireNonNull(exportService, \"exportService\");\n        this.workerExecutor = Objects.requireNonNull(workerExecutor, \"workerExecutor\");\n        this.ingestMetrics = Objects.requireNonNull(ingestMetrics, \"ingestMetrics\");\n        this.maxInFlightBytes = maxInFlightBytes;\n        this.admissionBytes = new Semaphore(maxInFlightBytes);\n        // reserved = budget - free permits; sampled per export step (see OtlpTraceIngestMetrics).\n        ingestMetrics.registerInFlightBytes(OtlpTraceIngestMetrics.Transport.GRPC,\n                () -> (long) maxInFlightBytes - admissionBytes.availablePermits(), maxInFlightBytes);\n    }\n\n    @Override\n    public void export(ExportTraceServiceRequest request, StreamObserver<ExportTraceServiceResponse> responseObserver) {\n        // Global byte-based admission: reserve this request's wire size from the in-flight budget\n        // before queuing. When the budget is exhausted, reject with UNAVAILABLE (retryable) so the\n        // exporter backs off, bounding total memory regardless of request size / connection count.\n        // (request size is already <= maxInboundMessageSize, enforced by gRPC before this handler.)\n        final int requestBytes = request.getSerializedSize();\n        if (!admissionBytes.tryAcquire(requestBytes)) {\n            logger.warn(\"Failed to export. In-flight byte budget exhausted. requestBytes={}, budget={}\", requestBytes, maxInFlightBytes);\n            ingestMetrics.requestRejected(OtlpTraceIngestMetrics.Transport.GRPC, OtlpTraceIngestMetrics.RequestRejectReason.INFLIGHT_BYTES);\n            safeOnError(responseObserver, ADMISSION_REJECTED);\n            return;\n        }\n\n        ingestMetrics.requestBytes(OtlpTraceIngestMetrics.Transport.GRPC, requestBytes);\n\n        final List<ResourceSpans> resourceSpanList = request.getResourceSpansList();\n        // Offload the mapping/insert work onto the worker pool so the gRPC handler thread\n        // (server.executor) is not blocked. A saturated worker queue surfaces to the client\n        // as UNAVAILABLE (retryable), providing backpressure instead of dropping spans.\n        final Context current = Context.current();\n        final Runnable exportTask = current.wrap(() -> {\n            try {\n                if (Context.current().isCancelled()) {\n                    // Client already gave up (deadline exceeded / cancelled) before this task ran.\n                    // Skip the wasted mapping/insert; the admission reservation is freed in finally.\n                    return;","sourceCodeStart":56,"sourceCodeEnd":92,"githubUrl":"https://github.com/pinpoint-apm/pinpoint/blob/744c3d3075e595656abb1ae331ad2c0e4c9eb996/otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/GrpcOtlpTraceService.java#L56-L92","documentation":"GrpcOtlpTraceService.export performs global byte-based admission: before queuing an OTLP trace export it calls admissionBytes.tryAcquire(request.getSerializedSize()) against a bounded in-flight byte budget. When the budget is exhausted the request is rejected with Status.UNAVAILABLE (retryable, so the OTLP exporter backs off), a warning is logged with the request size and budget, an INFLIGHT_BYTES rejection is recorded in metrics, and safeOnError returns ADMISSION_REJECTED to the client. This bounds collector memory regardless of request size or connection count.","triggerScenarios":"A gRPC Export call arrives while the sum of already-admitted in-flight serialized request bytes is within maxInflightBytes of the cap, so tryAcquire(requestBytes) fails. Requests larger than the remaining budget are rejected even if smaller ones would fit.","commonSituations":"Traffic spikes from many SDK instances, one very large batch consuming most of the budget, collector downscaling reducing capacity, or a client that ignores UNAVAILABLE and retries without backoff, keeping the budget saturated.","solutions":["Treat UNAVAILABLE as retryable: configure the OTLP exporter with exponential backoff and honor retry hints.","Reduce per-export payload size (smaller BatchSpanProcessor max_export_batch_size / max_queue_size).","Increase the collector's maxInFlightBytes admission budget or add collector replicas to spread load.","Inspect ingestMetrics INFLIGHT_BYTES rejection counts to size the budget against real peak traffic."],"exampleFix":"// before (client)\nOtlpGrpcSpanExporter.builder().build(); // default retry, immediate\n\n// after\nOtlpGrpcSpanExporter.builder()\n    .setTimeout(Duration.ofSeconds(10))\n    .build(); // SDK retries UNAVAILABLE with exponential backoff","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// gRPC UNAVAILABLE is the signal; enable SDK retry/backoff\nonError(throwable -> {\n    if (Status.fromThrowable(throwable).getCode() == Status.UNAVAILABLE.getCode()) {\n        scheduleRetryWithExponentialBackoff();\n    }\n});","preventionTips":["Always configure exponential backoff with jitter on the OTLP gRPC exporter.","Keep request size well below maxInboundMessageSize and the collector budget.","Scale collector replicas when INFLIGHT_BYTES rejections trend upward.","Avoid unbounded client-side concurrency against one collector."],"tags":["grpc","otlp","backpressure","unavailable","memory-limit"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"744c3d3075e595656abb1ae331ad2c0e4c9eb996","analyzedAt":"2026-09-07T18:48:45.289Z","contentChangedAt":"2026-09-07T18:48:45.289Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}