pinpoint-apm/pinpoint · error · StatusException

NOT_FOUND

NOT_FOUND

Error message

Could not find echo sink: clusterKey = {}

What it means

When the collector receives a command echo/activeThreadDump/activeThreadLightDump response from an agent, emitMono looks up the registered sink (Publisher) for that agent connection. If no sink is registered for the connection identified by the request's cluster key, it logs 'Could not find echo sink' and returns gRPC Status.NOT_FOUND to the caller. This means the collector has no active publisher session for that agent's command responses.

Source

Thrown at realtime/realtime-collector/src/main/java/com/navercorp/pinpoint/realtime/collector/receiver/grpc/GrpcCommandService.java:306

        long sinkId = response.getCommonResponse().getResponseId();
        final ActiveThreadLightDumpPublisher publisher = this.activeThreadLightDumpSinkRepo.get(sinkId);
        emitMono(response, responseObserver, publisher);
        this.activeThreadLightDumpSinkRepo.invalidate(sinkId);
    }

    @Override
    public StreamObserver<PCmdActiveThreadCountRes> commandStreamActiveThreadCount(StreamObserver<Empty> responseObserver) {
        logger.debug("commandStreamActiveThreadCount started");

        ServerCallStreamObserver<Empty> serverResponseObserver = (ServerCallStreamObserver<Empty>) responseObserver;
        return new ActiveThreadCountResponseStreamObserver(serverResponseObserver, this.activeThreadCountSinkRepo);
    }

    private <T> void emitMono(T response, StreamObserver<Empty> responseObserver, Publisher<T> sink) {
        if (sink == null) {
            if (logger.isWarnEnabled()) {
                Header header = ServerContext.getAgentInfo();
                logger.warn("Could not find echo sink: clusterKey = {}", getClusterKeyFromContext(header));
            }
            responseObserver.onError(new StatusException(Status.NOT_FOUND));
            return;
        }
        sink.publish(response);
        responseObserver.onNext(Empty.getDefaultInstance());
        responseObserver.onCompleted();
    }

    private InetSocketAddress getRemoteAddressFromContext() {
        TransportMetadata transportMetadata = ServerContext.getTransportMetadata();
        return transportMetadata.getRemoteAddress();
    }

    private ClusterKey getClusterKeyFromContext(Header header) {
        return new ClusterKey(header.getServiceName(), header.getApplicationName(), header.getAgentId(), header.getAgentStartTime());
    }

View on GitHub (pinned to 744c3d3075)

Solutions

1. Retry the command after confirming the agent is currently connected to the same collector instance that handles it
2. Ensure all collector nodes share the sink registry or that command routing is sticky to the node holding the agent's stream
3. Refresh the agent list in the web/cluster metadata so commands are not issued for stale agent sessions
4. Check collector logs for an earlier disconnect of the agent's command stream (which removes the sinks)

Example fix

// caller-side guard before invoking commandEcho
EchoPublisher sink = echoSinkRepo.getEchoPublisher(clusterKey);
if (sink == null) {
    // agent stream not connected to this collector; re-fetch or fail fast
    throw new AgentNotConnectedException(clusterKey);
}
profilerCommandService.commandEcho(responseObserver, echoRequest);

// before issuing a command, verify the sink exists on this collector
Publisher<?> sink = sinkRepository.getPublisher(clusterKey);
if (sink == null) {
    throw new AgentNotConnectedException("no sink registered for " + clusterKey);
}

try {
    profilerCommandService.commandEcho(responseObserver, echoRequest);
} catch (StatusRuntimeException e) {
    if (e.getStatus().getCode() == Status.Code.NOT_FOUND) {
        log.warn("Agent session gone; refresh cluster metadata and retry", e);
        retryAfterRefresh(echoRequest);
    }
}

When it happens

Trigger: A command response arrives from (or is emitted for) an agent whose sink was never created or was already removed from the SinkRepository — e.g. the agent's command stream disconnected and cleaned up sinks before the response was published, or the command was issued for an agent not currently connected to this collector instance.

Common situations: Web/collector topology mismatch: the command was sent through one collector node but the agent's stream is attached to another (no shared sink repo); agent reconnected between request and response; stale cluster metadata in the web UI issuing commands to a dead agent session.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• list size not same
• Failed to request. {} header={}
• Failed to request. header={}
• Failed to request. header={}
• PTransactionId is not set

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/b104930337721a88. Report an issue: GitHub.