pinpoint-apm/pinpoint · warning

dnsExecutor shutdown failed

Error message

dnsExecutor shutdown failed

What it means

During GrpcModuleLifeCycle.shutdown(), the dedicated DNS-resolution executor (dnsExecutorService, used by gRPC's DNS name resolver) is stopped with MoreExecutors.shutdownAndAwaitTermination with a 3-second timeout. If it does not terminate within 3 seconds, the code logs "dnsExecutor shutdown failed". It means lingering DNS-resolver threads may remain after shutdown — a thread-leak warning, not a fatal error.

Source

Thrown at agent-module/profiler/src/main/java/com/navercorp/pinpoint/profiler/context/module/GrpcModuleLifeCycle.java:116

        this.dnsExecutorService = dnsExecutorServiceProvider.get();
        logger.info("dnsExecutorServiceProvider:{}", dnsExecutorService);

        this.reconnectScheduledExecutorService = reconnectScheduledExecutorProvider.get();
        logger.info("reconnectScheduledExecutorServiceProvider:{}", reconnectScheduledExecutorService);

    }

    @Override
    public void shutdown() {
        logger.info("shutdown()");
        IOUtils.closeQuietly(spanDataSender, (ex) -> logger.warn("spanDataSender close fail", ex));
        IOUtils.closeQuietly(statDataSender, (ex) -> logger.warn("statDataSender close fail", ex));
        IOUtils.closeQuietly(agentDataSender, (ex) -> logger.warn("agentDataSender close fail", ex));
        IOUtils.closeQuietly(metadataDataSender, (ex) -> logger.warn("metadataDataSender close fail", ex));

        if (dnsExecutorService != null) {
            if (!MoreExecutors.shutdownAndAwaitTermination(dnsExecutorService, Duration.ofSeconds(3))) {
                logger.warn("dnsExecutor shutdown failed");
            }
        }
        if (reconnectScheduledExecutorService != null) {
            if (!MoreExecutors.shutdownAndAwaitTermination(reconnectScheduledExecutorService, Duration.ofSeconds(3))) {
                logger.warn("reconnectScheduledExecutor shutdown failed");
            }
        }
        IOUtils.closeQuietly(reporter, (ex) -> logger.warn("reporter close fail", ex));
    }

    @Override
    public String toString() {
        return "GrpcModuleLifeCycle{" +
                ", agentDataSender=" + agentDataSender +
                ", metadataDataSender=" + metadataDataSender +
                ", spanDataSender=" + spanDataSender +
                ", statDataSender=" + statDataSender +
                ", dnsExecutorService=" + dnsExecutorService +

View on GitHub (pinned to 744c3d3075)

Solutions

  1. Verify DNS server responsiveness (e.g. dig/nslookup from the agent host); fix resolver configuration in /etc/resolv.conf.
  2. Configure gRPC to use an IP address or a faster resolver for the collector address to reduce DNS dependency.
  3. Increase the 3s timeout in a patched GrpcModuleLifeCycle if your environment legitimately has slow DNS.
  4. Check for leaked pinpoint/dns threads after shutdown with jstack; if only a warning, leak is small and bounded.

Example fix

// before: hostname requires DNS lookup at shutdown
profiler.collector.span.ip=collector.pinpoint.local
// after: avoid runtime DNS dependency
profiler.collector.span.ip=10.0.0.5
Defensive patterns

Strategy: fallback

Validate before calling

// Pre-check DNS health before starting/stopping the agent
boolean dnsOk = InetAddress.getByName("collector.pinpoint.local") != null;

Try / catch

try {
    lifeCycle.shutdown();
} catch (Exception e) {
    logger.warn("shutdown warning ignored", e);
}

Prevention

When it happens

Trigger: shutdown() invoked while a DNS lookup is in progress or blocked (slow/unresponsive DNS server), so the executor's tasks ignore the shutdown signal and the 3s awaitTermination elapses.

Common situations: Corporate networks with slow DNS resolvers; DNS server outage while stopping the agent; containers where DNS queries hang; stopping the agent immediately after startup while the first resolution is still running.

Related errors

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/45c3ada30f3b4ee5. Report an issue: GitHub.