What causes "Failed async span insert (HBase). spanId={}" in pinpoint-apm/pinpoint?

Check the logged throwable for the HBase root cause and verify region server health at that timestamp.

Failed async span insert (HBase). spanId={}

Error message

Failed async span insert (HBase). spanId={}

What it means

HbaseOtlpTraceService.insertSpan submits an async HBase write of a SpanBo and registers a whenCompleteAsync callback; when the future completes exceptionally (the HBase async client failed to store the span), the error counter is incremented and this warning is logged with the spanId. The span data was received but never persisted.

Source

Thrown at otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/HbaseOtlpTraceService.java:91

    public void insertSpanChunk(SpanChunkBo spanChunkBo) {
        SpanChunkInsertEvent event = publisher.captureContext(spanChunkBo);
        traceDao.insertSpanChunk(spanChunkBo);
        applicationMapService.insertSpanChunk(spanChunkBo);
        publisher.publishEvent(event, true);
    }

    @Override
    public void insertSpan(SpanBo spanBo) {
        SpanInsertEvent event = publisher.captureContext(spanBo);
        CompletableFuture<Void> future = traceDao.asyncInsert(spanBo);
        scatterService.insert(spanBo);
        applicationMapService.insertSpan(spanBo);

        future.whenCompleteAsync((unused, throwable) -> {
            final boolean result = throwable == null;
            if (!result) {
                asyncInsertErrorCounter.increment();
                throttledLogger.warn("Failed async span insert (HBase). spanId={}", spanBo.getSpanId(), throwable);
            } else if (logger.isTraceEnabled()) {
                logger.trace("success {}", result);
            }
            publisher.publishEvent(event, result);
        }, grpcOtlpTraceServerExecutor);
    }
}

View on GitHub (pinned to 744c3d3075)

Solutions

Check the logged throwable for the HBase root cause and verify region server health at that timestamp.
Increase async client write timeouts / retries and tune executor queues in the trace DAO configuration.
Ensure the collector drains in-flight writes on shutdown instead of dropping them.
Scale collectors or shard span traffic if persistent write latency is causing timeouts.

Defensive patterns

Strategy: try-catch

Validate before calling

if (spanBo == null || spanBo.getSpanId() == -1) {
    throw new IllegalArgumentException("invalid SpanBo: spanId required before async insert");
}

Try / catch

future.whenCompleteAsync((unused, throwable) -> {
    if (throwable != null) {
        logger.warn("Failed async span insert (HBase). spanId={}", spanBo.getSpanId(), throwable);
        // inspect root cause: IOException -> HBase availability; TimeoutException -> tune timeouts
    }
}, grpcOtlpTraceServerExecutor);

Prevention

Monitor asyncInsertErrorCounter metric and alert on sustained growth.
Tune HBase async client write buffer and timeout settings for your span throughput.
Ensure graceful shutdown drains pending async writes before executor termination.
Keep region servers healthy; watch for hot-spotting on the trace table.

When it happens

Trigger: The AsyncFuture returned by the async trace DAO completes with a throwable — typically HBase write timeout, region unavailable, or client shutdown — during insertSpan(spanBo).

Common situations: HBase write queue backlog / hbase.client.write.buffer overflows under high span throughput; region server failover; collector shutting down while writes are in flight; HBase timeouts under GC pauses.

Related errors

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/7f7d3e44afaac347. Report an issue: GitHub.