What causes "Failed to insert spanBo" in pinpoint-apm/pinpoint?

Check the throttled-logged stack trace to see which TraceService threw.

Failed to insert spanBo

Error message

Failed to insert spanBo

What it means

OtlpTraceExportService.export loops over every configured TraceService calling insertSpan(spanBo); a synchronous exception from any of them (the OTLP-mapped span could not be handed to one sink) increments the insert error count, bumps spanInsertErrorCounter, and logs this throttled warning. The export continues with the remaining sinks but the failure is included in the returned serverErrorCount.

Source

Thrown at otlptrace/otlptrace-collector/src/main/java/com/navercorp/pinpoint/otlp/trace/collector/service/OtlpTraceExportService.java:150

        final OtlpTraceMapperData otlpTraceMapperData = otlpTraceMapper.map(resourceSpanList);
        // Mapping rejects (invalid ids, unlinkable/orphan spans) are client-side data faults.
        final OtlpTraceCollectorRejectedSpan clientRejected = otlpTraceMapperData.getRejectedSpan();
        for (Map.Entry<OtlpTraceRejectReason, Long> rejected : clientRejected.countByReason().entrySet()) {
            ingestMetrics.spanRejected(transport, rejected.getKey(), rejected.getValue());
        }
        ingestMetrics.spanStored(transport, otlpTraceMapperData.getSpanBoList().size());
        ingestMetrics.spanChunkStored(transport, otlpTraceMapperData.getSpanChunkBoList().size());

        int insertErrorCount = 0;
        for (SpanBo spanBo : otlpTraceMapperData.getSpanBoList()) {
            for (TraceService traceService : traceServiceList) {
                try {
                    traceService.insertSpan(spanBo);
                } catch (Exception e) {
                    insertErrorCount++;
                    spanInsertErrorCounter.increment();
                    throttledLogger.warn("Failed to insert spanBo", e);
                }
            }
        }

        for (SpanChunkBo spanChunkBo : otlpTraceMapperData.getSpanChunkBoList()) {
            for (TraceService traceService : traceServiceList) {
                try {
                    traceService.insertSpanChunk(spanChunkBo);
                } catch (Exception e) {
                    insertErrorCount++;
                    spanChunkInsertErrorCounter.increment();
                    throttledLogger.warn("Failed to insert spanChunkBo", e);
                }
            }
        }

        int agentInfoErrorCount = 0;
        for (AgentInfoBo agentInfoBo : otlpTraceMapperData.getAgentInfoBoList()) {

View on GitHub (pinned to 744c3d3075)

Solutions

Check the throttled-logged stack trace to see which TraceService threw.
Verify each configured trace sink's HBase tables and connectivity.
Inspect the export result's serverErrorCount/serverMessage returned to the ingest path for aggregate failure info.
Fix or disable the failing TraceService in the collector configuration.

Defensive patterns

Strategy: try-catch

Validate before calling

if (otlpTraceMapperData.getSpanBoList() == null || otlpTraceMapperData.getSpanBoList().isEmpty()) {
    return; // nothing to export, skip loop entirely
}

Try / catch

for (TraceService traceService : traceServiceList) {
    try {
        traceService.insertSpan(spanBo);
    } catch (Exception e) {
        spanInsertErrorCounter.increment();
        logger.warn("Failed to insert spanBo into {}", traceService.getClass().getSimpleName(), e);
    }
}

Prevention

Check the export result's serverErrorCount before confirming successful ingestion.
Verify every configured TraceService sink is reachable at collector startup.
Keep per-sink executor queues sized for peak span rates to avoid rejected submissions.
Alert on spanInsertErrorCounter rather than relying on throttled logs alone.

When it happens

Trigger: traceService.insertSpan(spanBo) throws synchronously for any registered TraceService while exporting mapped OTLP spans — e.g. HBase client exception, rejected executor submission, or mapping/data validation failure.

Common situations: A secondary trace sink (v1/v2 HBase writer) misconfigured or down; executor queue full under load spikes; malformed span after OTLP mapping.

Related errors

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/3f423c9de5df4659. Report an issue: GitHub.