What causes "spanDataSender close fail" in pinpoint-apm/pinpoint?

Inspect the nested exception in the log to identify the underlying channel failure

spanDataSender close fail

Error message

spanDataSender close fail

What it means

During GrpcModuleLifeCycle.shutdown(), each gRPC data sender (span, stat, agent, metadata) is closed via IOUtils.closeQuietly. When closing spanDataSender throws, the provided consumer logs the warning "spanDataSender close fail" — this is a logged warning, not an exception; shutdown continues with the remaining senders.

Source

Thrown at agent-module/profiler/src/main/java/com/navercorp/pinpoint/profiler/context/module/GrpcModuleLifeCycle.java:109

        this.spanDataSender = spanDataSenderProvider.get();
        logger.info("spanDataSenderProvider:{}", spanDataSender);

        this.statDataSender = this.statDataSenderProvider.get();
        logger.info("statDataSenderProvider:{}", statDataSender);

        this.dnsExecutorService = dnsExecutorServiceProvider.get();
        logger.info("dnsExecutorServiceProvider:{}", dnsExecutorService);

        this.reconnectScheduledExecutorService = reconnectScheduledExecutorProvider.get();
        logger.info("reconnectScheduledExecutorServiceProvider:{}", reconnectScheduledExecutorService);

    }

    @Override
    public void shutdown() {
        logger.info("shutdown()");
        IOUtils.closeQuietly(spanDataSender, (ex) -> logger.warn("spanDataSender close fail", ex));
        IOUtils.closeQuietly(statDataSender, (ex) -> logger.warn("statDataSender close fail", ex));
        IOUtils.closeQuietly(agentDataSender, (ex) -> logger.warn("agentDataSender close fail", ex));
        IOUtils.closeQuietly(metadataDataSender, (ex) -> logger.warn("metadataDataSender close fail", ex));

        if (dnsExecutorService != null) {
            if (!MoreExecutors.shutdownAndAwaitTermination(dnsExecutorService, Duration.ofSeconds(3))) {
                logger.warn("dnsExecutor shutdown failed");
            }
        }
        if (reconnectScheduledExecutorService != null) {
            if (!MoreExecutors.shutdownAndAwaitTermination(reconnectScheduledExecutorService, Duration.ofSeconds(3))) {
                logger.warn("reconnectScheduledExecutor shutdown failed");
            }
        }
        IOUtils.closeQuietly(reporter, (ex) -> logger.warn("reporter close fail", ex));
    }

    @Override

View on GitHub (pinned to 744c3d3075)

Solutions

Inspect the nested exception in the log to identify the underlying channel failure
Verify network connectivity/DNS to the collector at shutdown time
Ensure shutdown() is called only once per module lifecycle; double shutdown can surface as close failures
If errors are purely during shutdown of an already-broken connection and the process is exiting, they are safe to tolerate

Defensive patterns

Strategy: try-catch

Validate before calling

if (channel != null && channel.isShutdown()) { logger.warn("span channel already closed; skip graceful close"); }

Type guard

boolean isClosable(Object sender) { return sender instanceof io.grpc.ManagedChannel && !((io.grpc.ManagedChannel) sender).isShutdown(); }

Try / catch

try {
  profiler.stop(); // triggers GrpcModuleLifeCycle.shutdown()
} catch (Throwable t) {
  logger.warn("Shutdown reported sender close problems", t); // individual sender failures are already logged as warnings
}

Prevention

Ensure the collector is reachable when performing graceful shutdowns
Avoid double shutdown; guard lifecycle state before calling stop()
Increase shutdown hooks' patience so gRPC close() can finish (avoid SIGKILL)

When it happens

Trigger: Calling profiler shutdown while spanDataSender's underlying gRPC channel/StreamObserver is in a failed state (network already down, channel already closed, in-flight RPCs failing on close).

Common situations: Agent shutdown during network outage to the collector, double shutdown of the module, abrupt termination (kill) leaving the channel half-closed, collector stopped before agent shutdown.

Related errors

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/ff1b65da84c976a0. Report an issue: GitHub.