{"record":{"id":"906972725d7ff31d","repo":"quickwit-oss/quickwit","slug":"could-not-find-any-service-node-in-the-cluster","errorCode":null,"errorMessage":"could not find any `{service}` node in the cluster","messagePattern":"could not find any `(.+?)` node in the cluster","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-serve/src/metastore.rs","lineNumber":164,"sourceCode":"                    node_config.grpc_config.max_message_size,\n                )\n                .await?;\n                Ok(Some(read_replica_client))\n            }\n            LocalMetastoreServer::Primary(_) | LocalMetastoreServer::NotServed => Ok(None),\n        }\n    }\n\n    async fn build_metastore_client(\n        cluster: &Cluster,\n        service: QuickwitService,\n        max_message_size: ByteSize,\n    ) -> anyhow::Result<MetastoreServiceClient> {\n        info!(%service, \"connecting to {service} service\");\n\n        let balance_channel = balance_channel_for_service(cluster, service).await;\n\n        ensure!(\n            balance_channel\n                .wait_for(Duration::from_mins(5), |connections| {\n                    !connections.is_empty()\n                })\n                .await,\n            \"could not find any `{service}` node in the cluster\"\n        );\n        let metrics_layer = match service {\n            QuickwitService::Metastore => PRIMARY_METASTORE_GRPC_CLIENT_METRICS_LAYER.clone(),\n            QuickwitService::MetastoreReadReplica => {\n                READ_REPLICA_METASTORE_GRPC_CLIENT_METRICS_LAYER.clone()\n            }\n            _ => unreachable!(\"unexpected metastore service `{service}`\"),\n        };\n        let retry_policy =\n            metrics_layer.from_retry_policy(RetryPolicy::from(RetryParams::standard()));\n        Ok(MetastoreServiceClient::tower()\n            // Metrics wrap retries so they record only the final outcome.","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-serve/src/metastore.rs#L146-L182","documentation":"When building a MetastoreServiceClient for a gRPC service, the code waits (up to 5 minutes) for the cluster's load-balancing channel to have at least one connection to a node advertising that service. If no node in the cluster runs the requested service (e.g. `metastore`), the wait times out and this error is thrown.","triggerScenarios":"Calling `build_metastore_client(cluster, service, ...)` in a cluster where no chitchat peer publishes readiness for the given service — e.g. running the searcher/ingester while no node has the `metastore` service enabled, or the node is slow/unreachable so the 5-minute wait expires.","commonSituations":"Single-binary mode where the metastore service isn't enabled; misconfigured `service` list in node config; cluster partition or slow startup; wrong cluster name so the client sees no peers.","solutions":["Enable the target service on at least one node (`service: [metastore, ...]` in node config or `--service metastore`).","Verify all nodes join the same cluster (matching `cluster_id`/advertise address) and are reachable.","Check node logs/chitchat status to confirm the service node is up and has published readiness."],"exampleFix":"// before (quickwit.yaml)\nnode:\n  service: [searcher]\n// after\nnode:\n  service: [searcher, metastore]","handlingStrategy":"retry","validationCode":"// before connecting, check chitchat membership for the service:\n// cluster.ready_nodes_for_service(service).is_empty() => enable the service or wait","typeGuard":null,"tryCatchPattern":"loop {\n    match build_metastore_client(&cluster, service, timeout, max_message_size).await {\n        Ok(client) => break client,\n        Err(e) if e.to_string().contains(\"could not find any\") => {\n            tokio::time::sleep(Duration::from_secs(10)).await; // node may still be starting\n        }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Ensure at least one node lists the service (e.g. metastore) in its `service` config.","Verify cluster_id and network reachability between nodes before startup.","Monitor chitchat readiness so missing-service nodes are alerted early."],"tags":["cluster","grpc","service-discovery"],"backgroundTag":"resource-not-found","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}