{"record":{"id":"7f7d3e44afaac347","repo":"pinpoint-apm/pinpoint","slug":"failed-async-span-insert-hbase-spanid","errorCode":null,"errorMessage":"Failed async span insert (HBase). spanId={}","messagePattern":"Failed async span insert \\(HBase\\)\\. spanId=(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/HbaseOtlpTraceService.java","lineNumber":91,"sourceCode":"    public void insertSpanChunk(SpanChunkBo spanChunkBo) {\n        SpanChunkInsertEvent event = publisher.captureContext(spanChunkBo);\n        traceDao.insertSpanChunk(spanChunkBo);\n        applicationMapService.insertSpanChunk(spanChunkBo);\n        publisher.publishEvent(event, true);\n    }\n\n    @Override\n    public void insertSpan(SpanBo spanBo) {\n        SpanInsertEvent event = publisher.captureContext(spanBo);\n        CompletableFuture<Void> future = traceDao.asyncInsert(spanBo);\n        scatterService.insert(spanBo);\n        applicationMapService.insertSpan(spanBo);\n\n        future.whenCompleteAsync((unused, throwable) -> {\n            final boolean result = throwable == null;\n            if (!result) {\n                asyncInsertErrorCounter.increment();\n                throttledLogger.warn(\"Failed async span insert (HBase). spanId={}\", spanBo.getSpanId(), throwable);\n            } else if (logger.isTraceEnabled()) {\n                logger.trace(\"success {}\", result);\n            }\n            publisher.publishEvent(event, result);\n        }, grpcOtlpTraceServerExecutor);\n    }\n}\n","sourceCodeStart":73,"sourceCodeEnd":99,"githubUrl":"https://github.com/pinpoint-apm/pinpoint/blob/744c3d3075e595656abb1ae331ad2c0e4c9eb996/otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/HbaseOtlpTraceService.java#L73-L99","documentation":"HbaseOtlpTraceService.insertSpan submits an async HBase write of a SpanBo and registers a whenCompleteAsync callback; when the future completes exceptionally (the HBase async client failed to store the span), the error counter is incremented and this warning is logged with the spanId. The span data was received but never persisted.","triggerScenarios":"The AsyncFuture returned by the async trace DAO completes with a throwable — typically HBase write timeout, region unavailable, or client shutdown — during insertSpan(spanBo).","commonSituations":"HBase write queue backlog / hbase.client.write.buffer overflows under high span throughput; region server failover; collector shutting down while writes are in flight; HBase timeouts under GC pauses.","solutions":["Check the logged throwable for the HBase root cause and verify region server health at that timestamp.","Increase async client write timeouts / retries and tune executor queues in the trace DAO configuration.","Ensure the collector drains in-flight writes on shutdown instead of dropping them.","Scale collectors or shard span traffic if persistent write latency is causing timeouts."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (spanBo == null || spanBo.getSpanId() == -1) {\n    throw new IllegalArgumentException(\"invalid SpanBo: spanId required before async insert\");\n}","typeGuard":null,"tryCatchPattern":"future.whenCompleteAsync((unused, throwable) -> {\n    if (throwable != null) {\n        logger.warn(\"Failed async span insert (HBase). spanId={}\", spanBo.getSpanId(), throwable);\n        // inspect root cause: IOException -> HBase availability; TimeoutException -> tune timeouts\n    }\n}, grpcOtlpTraceServerExecutor);","preventionTips":["Monitor asyncInsertErrorCounter metric and alert on sustained growth.","Tune HBase async client write buffer and timeout settings for your span throughput.","Ensure graceful shutdown drains pending async writes before executor termination.","Keep region servers healthy; watch for hot-spotting on the trace table."],"tags":["hbase","async-write","span","timeout"],"backgroundTag":"database-write-failed","analyzedSha":"744c3d3075e595656abb1ae331ad2c0e4c9eb996","analyzedAt":"2026-09-07T18:48:45.289Z","contentChangedAt":"2026-09-07T18:48:45.289Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}