databendlabs/databend · error · ConnectionRefused

in-process gRPC endpoint closed

Error message

in-process gRPC endpoint closed

What it means

A sentinel error surfaced in connect_in_process_grpc when the duplex channel to the in-process gRPC endpoint fails — the registered in-process endpoint's sender has been dropped (server shut down) so no connection can be established. It is a generic guard: the target address has no live in-process receiver, meaning the meta or service process hosting the endpoint is not running in this process.

Solutions

1. Ensure the in-process gRPC endpoint is registered before calling connect, and that it is not shut down while clients are still connecting.
2. Re-register the endpoint or retry connect once registration completes.
3. Audit lifecycle ordering so the endpoint handle outlives all expected client connections.

When it happens

Trigger: Thrown at src/meta/runtime/src/in_process_grpc.rs:189 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/31b7f749152a027e. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/runtime/src/in_process_grpc.rs:189

        let sender = sender.ok_or_else(|| ChannelError::CannotConnect {
            uri: addr.clone(),
            message: "in-process gRPC endpoint is not registered".to_string(),
        })?;

        let mut endpoint = Endpoint::from_static("http://[::]:0");
        if let Some(timeout) = timeout {
            endpoint = endpoint.timeout(timeout);
        }

        let connector = service_fn(move |_| {
            let sender = sender.clone();
            async move {
                let (client, server) = tokio::io::duplex(IN_PROCESS_GRPC_BUFFER_SIZE);
                sender
                    .send(InProcessGrpcStream { inner: server })
                    .map_err(|_| {
                        io::Error::new(
                            io::ErrorKind::ConnectionRefused,
                            "in-process gRPC endpoint closed",
                        )
                    })?;
                Ok::<_, io::Error>(TokioIo::new(client))
            }
        });

        endpoint
            .connect_with_connector(connector)
            .await
            .map_err(|e| ChannelError::CannotConnect {
                uri: addr,
                message: e.to_string(),
            })
    })
}

View on GitHub (pinned to 288d84d76e)