quickwit-oss/quickwit · error

could not find any `{service}` node in the cluster

Error message

could not find any `{service}` node in the cluster

What it means

When building a MetastoreServiceClient for a gRPC service, the code waits (up to 5 minutes) for the cluster's load-balancing channel to have at least one connection to a node advertising that service. If no node in the cluster runs the requested service (e.g. `metastore`), the wait times out and this error is thrown.

Source

Thrown at quickwit/quickwit-serve/src/metastore.rs:164

                    node_config.grpc_config.max_message_size,
                )
                .await?;
                Ok(Some(read_replica_client))
            }
            LocalMetastoreServer::Primary(_) | LocalMetastoreServer::NotServed => Ok(None),
        }
    }

    async fn build_metastore_client(
        cluster: &Cluster,
        service: QuickwitService,
        max_message_size: ByteSize,
    ) -> anyhow::Result<MetastoreServiceClient> {
        info!(%service, "connecting to {service} service");

        let balance_channel = balance_channel_for_service(cluster, service).await;

        ensure!(
            balance_channel
                .wait_for(Duration::from_mins(5), |connections| {
                    !connections.is_empty()
                })
                .await,
            "could not find any `{service}` node in the cluster"
        );
        let metrics_layer = match service {
            QuickwitService::Metastore => PRIMARY_METASTORE_GRPC_CLIENT_METRICS_LAYER.clone(),
            QuickwitService::MetastoreReadReplica => {
                READ_REPLICA_METASTORE_GRPC_CLIENT_METRICS_LAYER.clone()
            }
            _ => unreachable!("unexpected metastore service `{service}`"),
        };
        let retry_policy =
            metrics_layer.from_retry_policy(RetryPolicy::from(RetryParams::standard()));
        Ok(MetastoreServiceClient::tower()
            // Metrics wrap retries so they record only the final outcome.

View on GitHub (pinned to a39730c5cd)

Solutions

1. Enable the target service on at least one node (`service: [metastore, ...]` in node config or `--service metastore`).
2. Verify all nodes join the same cluster (matching `cluster_id`/advertise address) and are reachable.
3. Check node logs/chitchat status to confirm the service node is up and has published readiness.

Example fix

// before (quickwit.yaml)
node:
  service: [searcher]
// after
node:
  service: [searcher, metastore]

// before connecting, check chitchat membership for the service:
// cluster.ready_nodes_for_service(service).is_empty() => enable the service or wait

loop {
    match build_metastore_client(&cluster, service, timeout, max_message_size).await {
        Ok(client) => break client,
        Err(e) if e.to_string().contains("could not find any") => {
            tokio::time::sleep(Duration::from_secs(10)).await; // node may still be starting
        }
        Err(e) => return Err(e),
    }
}

When it happens

Trigger: Calling `build_metastore_client(cluster, service, ...)` in a cluster where no chitchat peer publishes readiness for the given service — e.g. running the searcher/ingester while no node has the `metastore` service enabled, or the node is slow/unreachable so the 5-minute wait expires.

Common situations: Single-binary mode where the metastore service isn't enabled; misconfigured `service` list in node config; cluster partition or slow startup; wrong cluster name so the client sees no peers.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• compactor is enabled but no janitor node was found in the cl
• could not find control plane in the cluster
• timed out while waiting for ingester to transition to status
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/906972725d7ff31d. Report an issue: GitHub.