{"record":{"id":"79df4d32e02bd975","repo":"pinpoint-apm/pinpoint","slug":"exception-occurred-while-request-message-message","errorCode":null,"errorMessage":"Exception occurred while request message. message:{}","messagePattern":"Exception occurred while request message\\. message:(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"realtime/realtime-collector/src/main/java/com/navercorp/pinpoint/realtime/collector/controller/ClusterPointController.java","lineNumber":186,"sourceCode":"\n            if (startTimestamp > 0 && destClusterInfo.getStartTimestamp() != startTimestamp) {\n                continue;\n            }\n\n            result.add((GrpcAgentConnection) clusterPoint);\n        }\n\n        return result;\n    }\n\n    private CheckConnectionStatusResult request(GrpcAgentConnection grpcAgentConnection, int checkCount) {\n        logger.info(\"Ping  message will be sent. collector => {}.\", grpcAgentConnection.getClusterKey());\n\n        CompletableFuture<PCmdEchoResponse> response = null;\n        try {\n            response = request0(grpcAgentConnection, checkCount);\n        } catch (StatusRuntimeException e) {\n            logger.warn(\"Exception occurred while request message. message:{}\", e.getMessage(), e);\n            if (e.getStatus().getCode() == Status.CANCELLED.getCode()) {\n                clearConnection();\n                return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION;\n            }\n            return CheckConnectionStatusResult.FAIL;\n        } catch (ClusterException e) {\n            logger.warn(\"Exception occurred while request message. message:{}\", e.getMessage(), e);\n            clearConnection();\n            return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION;\n        }\n        try {\n            PCmdEchoResponse result = response.get(3000, TimeUnit.MILLISECONDS);\n            if (result.getMessage().equals(\"PING\")) {\n                return CheckConnectionStatusResult.SUCCESS;\n            }\n            logger.warn(\"Receive unexpected response: result = {}\", result);\n        } catch (Exception cause) {\n            logger.warn(\"Failed while request message. message:{}\", cause.getMessage(), cause);","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/pinpoint-apm/pinpoint/blob/744c3d3075e595656abb1ae331ad2c0e4c9eb996/realtime/realtime-collector/src/main/java/com/navercorp/pinpoint/realtime/collector/controller/ClusterPointController.java#L168-L204","documentation":"ClusterPointController.request sends a ping/echo command to a pinpoint agent over an active gRPC agent connection; when the gRPC call throws StatusRuntimeException this warning is logged with the status message. A CANCELLED status triggers clearConnection and FAIL_AND_CLEAR_CONNECTION so the stale agent connection is removed.","triggerScenarios":"request0(grpcAgentConnection, checkCount) throws StatusRuntimeException — the gRPC stream is dead/cancelled, the remote agent disconnected, DEADLINE_EXCEEDED, or UNAVAILABLE while sending the echo command.","commonSituations":"Agent restarted or network dropped but collector connection state is stale; gRPC channel idle-timed out; load balancer silently dropping long-lived streams; agent busy and stream cancelled.","solutions":["Check the logged status code — CANCELLED/UNAVAILABLE indicates a stale or broken channel.","Rely on the built-in clearConnection to remove dead connections and let the agent reconnect.","Verify agent-to-collector network stability (keepalive settings, LB idle timeouts) for long-lived gRPC streams.","Configure gRPC keepalive pings so half-open connections are detected earlier than the ping check."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (grpcAgentConnection == null || grpcAgentConnection.getClusterKey() == null) {\n    return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION; // nothing to ping\n}","typeGuard":null,"tryCatchPattern":"try {\n    response = request0(grpcAgentConnection, checkCount);\n} catch (StatusRuntimeException e) {\n    if (e.getStatus().getCode() == Status.CANCELLED.getCode()) {\n        clearConnection();\n        return CheckConnectionStatusResult.FAIL_AND_CLEAR_CONNECTION;\n    }\n    logger.warn(\"Exception occurred while request message. message:{}\", e.getMessage(), e);\n    return CheckConnectionStatusResult.FAIL;\n}","preventionTips":["Configure gRPC keepalive so stale channels are detected before ping checks.","Ensure load balancers do not silently kill long-lived agent streams (idle timeout tuning).","Rely on clearConnection + agent re-registration to self-heal stale connections.","Track StatusRuntimeException rates per clusterKey to spot flaky agents/hosts."],"tags":["grpc","agent-connection","ping","realtime"],"backgroundTag":"connection-refused","analyzedSha":"744c3d3075e595656abb1ae331ad2c0e4c9eb996","analyzedAt":"2026-09-07T18:48:45.289Z","contentChangedAt":"2026-09-07T18:48:45.289Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}