quickwit-oss/quickwit · error

could not find control plane in the cluster

Error message

could not find control plane in the cluster

What it means

When a node is not configured to run the control plane itself, serve_quickwit waits up to 5 minutes on the gRPC balance channel for a control-plane node to appear via service discovery. If no control-plane connection is found within that window, it bails with 'could not find control plane in the cluster'.

Source

Thrown at quickwit/quickwit-serve/src/lib.rs:467

            balance_channel_for_service(cluster, QuickwitService::ControlPlane).await;

        // If the node is a metastore, we skip this check in order to avoid a deadlock.
        // A read-replica metastore node is skipped for the same reason: it only serves read-only
        // metastore traffic and does not need the control plane.
        // If the node is a searcher, we skip this check because the searcher does not need to.
        if !node_config.is_service_enabled(QuickwitService::Metastore)
            && !node_config.is_service_enabled(QuickwitService::MetastoreReadReplica)
            && node_config.enabled_services != HashSet::from([QuickwitService::Searcher])
        {
            info!("connecting to control plane");

            if !balance_channel
                .wait_for(Duration::from_mins(5), |connections| {
                    !connections.is_empty()
                })
                .await
            {
                bail!("could not find control plane in the cluster");
            }
        }
        let control_plane_server_opt = None;
        let control_plane_client = ControlPlaneServiceClient::tower()
            .stack_layer(CP_GRPC_CLIENT_METRICS_LAYER.clone())
            .build_from_balance_channel(
                balance_channel,
                node_config.grpc_config.max_message_size,
                None,
            );
        Ok((control_plane_server_opt, control_plane_client))
    }
}

fn start_shard_positions_service(
    ingester_opt: Option<Ingester>,
    cluster: Cluster,
    event_broker: EventBroker,

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure exactly the intended nodes run the control plane role and that it is up (check logs and Chitchat cluster membership).
2. Verify peer_seeds/cluster configuration so this node joins the same cluster as the control plane.
3. Check gRPC connectivity and firewall rules between the node and the control-plane node.
4. Fix the control-plane startup failure first, then restart this node; or enable the control plane role on this node for single-node setups.

Example fix

// before (searcher-only node, no control plane anywhere)
roles: [searcher]
// after (add control plane on a dedicated node or same node)
roles: [searcher, control-plane]

// Before starting a non-control-plane node, confirm the control plane is reachable
const members = await fetchClusterMembers();
if (!members.some(m => m.roles.includes('control-plane') && m.status === 'alive')) {
  throw new Error('Control plane node must be running before starting searcher/indexer');
}

function hasAliveControlPlane(members) {
  return Array.isArray(members) && members.some(
    m => Array.isArray(m.roles) && m.roles.includes('control-plane') && m.status === 'alive'
  );
}

try {
  await serveQuickwit(config);
} catch (e) {
  if (e.message.includes('could not find control plane')) {
    console.error('Start/control the control-plane node, verify peer_seeds, then retry.');
  }
  process.exit(1);
}

When it happens

Trigger: Starting a searcher/indexer/compactor node with the control-plane role disabled on that node while no other node in the cluster runs the control plane, or the control-plane node is down/unreachable and does not register its gRPC service within the 5-minute wait.

Common situations: Split-role deployments where the control-plane node failed to start or crashed; networking between nodes blocked; node started before the control plane ever deployed; Chitchat membership misconfigured (different cluster IDs or peer addresses) so the service is never discovered.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• compactor is enabled but no janitor node was found in the cl
• could not find any `{service}` node in the cluster
• timed out while waiting for ingester to transition to status
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/f768c0cb00320ce7. Report an issue: GitHub.