{"record":{"id":"f768c0cb00320ce7","repo":"quickwit-oss/quickwit","slug":"could-not-find-control-plane-in-the-cluster","errorCode":null,"errorMessage":"could not find control plane in the cluster","messagePattern":"could not find control plane in the cluster","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-serve/src/lib.rs","lineNumber":467,"sourceCode":"            balance_channel_for_service(cluster, QuickwitService::ControlPlane).await;\n\n        // If the node is a metastore, we skip this check in order to avoid a deadlock.\n        // A read-replica metastore node is skipped for the same reason: it only serves read-only\n        // metastore traffic and does not need the control plane.\n        // If the node is a searcher, we skip this check because the searcher does not need to.\n        if !node_config.is_service_enabled(QuickwitService::Metastore)\n            && !node_config.is_service_enabled(QuickwitService::MetastoreReadReplica)\n            && node_config.enabled_services != HashSet::from([QuickwitService::Searcher])\n        {\n            info!(\"connecting to control plane\");\n\n            if !balance_channel\n                .wait_for(Duration::from_mins(5), |connections| {\n                    !connections.is_empty()\n                })\n                .await\n            {\n                bail!(\"could not find control plane in the cluster\");\n            }\n        }\n        let control_plane_server_opt = None;\n        let control_plane_client = ControlPlaneServiceClient::tower()\n            .stack_layer(CP_GRPC_CLIENT_METRICS_LAYER.clone())\n            .build_from_balance_channel(\n                balance_channel,\n                node_config.grpc_config.max_message_size,\n                None,\n            );\n        Ok((control_plane_server_opt, control_plane_client))\n    }\n}\n\nfn start_shard_positions_service(\n    ingester_opt: Option<Ingester>,\n    cluster: Cluster,\n    event_broker: EventBroker,","sourceCodeStart":449,"sourceCodeEnd":485,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-serve/src/lib.rs#L449-L485","documentation":"When a node is not configured to run the control plane itself, serve_quickwit waits up to 5 minutes on the gRPC balance channel for a control-plane node to appear via service discovery. If no control-plane connection is found within that window, it bails with 'could not find control plane in the cluster'.","triggerScenarios":"Starting a searcher/indexer/compactor node with the control-plane role disabled on that node while no other node in the cluster runs the control plane, or the control-plane node is down/unreachable and does not register its gRPC service within the 5-minute wait.","commonSituations":"Split-role deployments where the control-plane node failed to start or crashed; networking between nodes blocked; node started before the control plane ever deployed; Chitchat membership misconfigured (different cluster IDs or peer addresses) so the service is never discovered.","solutions":["Ensure exactly the intended nodes run the control plane role and that it is up (check logs and Chitchat cluster membership).","Verify peer_seeds/cluster configuration so this node joins the same cluster as the control plane.","Check gRPC connectivity and firewall rules between the node and the control-plane node.","Fix the control-plane startup failure first, then restart this node; or enable the control plane role on this node for single-node setups."],"exampleFix":"// before (searcher-only node, no control plane anywhere)\nroles: [searcher]\n// after (add control plane on a dedicated node or same node)\nroles: [searcher, control-plane]","handlingStrategy":"validation","validationCode":"// Before starting a non-control-plane node, confirm the control plane is reachable\nconst members = await fetchClusterMembers();\nif (!members.some(m => m.roles.includes('control-plane') && m.status === 'alive')) {\n  throw new Error('Control plane node must be running before starting searcher/indexer');\n}","typeGuard":"function hasAliveControlPlane(members) {\n  return Array.isArray(members) && members.some(\n    m => Array.isArray(m.roles) && m.roles.includes('control-plane') && m.status === 'alive'\n  );\n}","tryCatchPattern":"try {\n  await serveQuickwit(config);\n} catch (e) {\n  if (e.message.includes('could not find control plane')) {\n    console.error('Start/control the control-plane node, verify peer_seeds, then retry.');\n  }\n  process.exit(1);\n}","preventionTips":["Use ordered deployment: control plane first, wait for liveness, then searcher/indexer/compactor nodes.","Verify peer_seeds/cluster_id match across all nodes so discovery works.","Alert on control-plane process death; it is a hard dependency for other roles.","For dev/single-node environments, enable the control-plane role locally."],"tags":["grpc","service-discovery","control-plane","cluster"],"backgroundTag":"resource-not-found","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}