What causes "Exception occurred while request message. message:{}" in pinpoint-apm/pinpoint?

Check the logged status code — CANCELLED/UNAVAILABLE indicates a stale or broken channel.

Exception occurred while request message. message

Error message

Exception occurred while request message. message:{}

What it means

ClusterPointController.request sends a ping/echo command to a pinpoint agent over an active gRPC agent connection; when the gRPC call throws StatusRuntimeException this warning is logged with the status message. A CANCELLED status triggers clearConnection and FAIL_AND_CLEAR_CONNECTION so the stale agent connection is removed.

Solutions

Check the logged status code — CANCELLED/UNAVAILABLE indicates a stale or broken channel.
Rely on the built-in clearConnection to remove dead connections and let the agent reconnect.
Verify agent-to-collector network stability (keepalive settings, LB idle timeouts) for long-lived gRPC streams.
Configure gRPC keepalive pings so half-open connections are detected earlier than the ping check.

Defensive patterns

Strategy: try-catch

Validate before calling

if (grpcAgentConnection == null || grpcAgentConnection.getClusterKey() == null) {
    return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION; // nothing to ping
}

Try / catch

try {
    response = request0(grpcAgentConnection, checkCount);
} catch (StatusRuntimeException e) {
    if (e.getStatus().getCode() == Status.CANCELLED.getCode()) {
        clearConnection();
        return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION;
    }
    logger.warn("Exception occurred while request message. message:{}", e.getMessage(), e);
    return CheckConnectionStatusResult.FAIL;
}

Prevention

Configure gRPC keepalive so stale channels are detected before ping checks.
Ensure load balancers do not silently kill long-lived agent streams (idle timeout tuning).
Rely on clearConnection + agent re-registration to self-heal stale connections.
Track StatusRuntimeException rates per clusterKey to spot flaky agents/hosts.

When it happens

Trigger: request0(grpcAgentConnection, checkCount) throws StatusRuntimeException — the gRPC stream is dead/cancelled, the remote agent disconnected, DEADLINE_EXCEEDED, or UNAVAILABLE while sending the echo command.

Common situations: Agent restarted or network dropped but collector connection state is stale; gRPC channel idle-timed out; load balancer silently dropping long-lived streams; agent busy and stream cancelled.

Understand the failure class

Background: ECONNREFUSED and "connection refused" / "could not connect to server" errors: what they mean and how to fix them — this error's family across 44 libraries.

Related errors

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/79df4d32e02bd975. Report an issue: GitHub.

Appendix: source

Thrown at realtime/realtime-collector/src/main/java/com/navercorp/pinpoint/realtime/collector/controller/ClusterPointController.java:186

            if (startTimestamp > 0 && destClusterInfo.getStartTimestamp() != startTimestamp) {
                continue;
            }

            result.add((GrpcAgentConnection) clusterPoint);
        }

        return result;
    }

    private CheckConnectionStatusResult request(GrpcAgentConnection grpcAgentConnection, int checkCount) {
        logger.info("Ping  message will be sent. collector => {}.", grpcAgentConnection.getClusterKey());

        CompletableFuture<PCmdEchoResponse> response = null;
        try {
            response = request0(grpcAgentConnection, checkCount);
        } catch (StatusRuntimeException e) {
            logger.warn("Exception occurred while request message. message:{}", e.getMessage(), e);
            if (e.getStatus().getCode() == Status.CANCELLED.getCode()) {
                clearConnection();
                return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION;
            }
            return CheckConnectionStatusResult.FAIL;
        } catch (ClusterException e) {
            logger.warn("Exception occurred while request message. message:{}", e.getMessage(), e);
            clearConnection();
            return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION;
        }
        try {
            PCmdEchoResponse result = response.get(3000, TimeUnit.MILLISECONDS);
            if (result.getMessage().equals("PING")) {
                return CheckConnectionStatusResult.SUCCESS;
            }
            logger.warn("Receive unexpected response: result = {}", result);
        } catch (Exception cause) {
            logger.warn("Failed while request message. message:{}", cause.getMessage(), cause);

View on GitHub (pinned to 744c3d3075)