databendlabs/databend · error

leader node not found

Error message

leader node {} not found

What it means

In do_register, when the node being registered is not handled locally, the request is forwarded to the current raft leader over gRPC. The code first looks the leader up via meta_handle.handle_get_node(leader_id); if that returns None it raises this error because the leader's endpoint cannot be resolved. Without the endpoint the forwarding cannot proceed.

Solutions

1. Retry registration after a short delay — membership usually converges and handle_get_node will then find the leader.
2. Verify cluster node list (admin API / nodes table) contains the node with leader_id and correct raft endpoint.
3. Check raft logs for recent leader changes or snapshot/compaction issues that could desynchronize the node registry.
4. If the leader record is genuinely gone, re-register/repair cluster membership (re-add the node) before forwarding registration.

Example fix

// before
.ok_or_else(|| anyhow::anyhow!("leader node {} not found", leader_id))?;
// after
match meta_handle.handle_get_node(leader_id).await? {
    Some(node) => node.endpoint,
    None => return Err(anyhow::anyhow!(
        "leader node {} not found; cluster membership may be converging, retry", leader_id)),
}

// Rust: confirm the leader exists before forwarding
let known = meta_handle.handle_get_node(leader_id).await?.is_some();
if !known { /* delay and re-read raft state before registering */ }

let leader = retry(3, backoff, || async {
    meta_handle.handle_get_node(leader_id).await?
        .ok_or_else(|| anyhow!("leader {leader_id} not yet visible; retrying"))
}).await?;

When it happens

Trigger: register/leave requests arriving while cluster membership metadata is stale: leader_id from raft state refers to a node absent from the node registry (registry not yet synced after leader change, or the leader node was removed/de-registered).

Common situations: Right after a leader election where the new leader's node record hasn't propagated; operating on a cluster whose node list was manually edited; a joining node contacting an out-of-date follower.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• timeout; no error received
• Unsupported format for
• Unsupported source type. Expected path, pandas.DataFrame…
• Access denied: is outside allowed directories
• rename_database: src (db) should exist

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/aac64cfa0929ba6d. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/binaries/meta/entry.rs:264

    info!("Raft log entry for updating node: {:?}", ent);

    if meta_handle.id == leader_id {
        // We are the leader: write directly via raft to avoid stale endpoint lookup.
        info!("This node is the leader, writing directly");
        meta_handle
            .request(move |meta_node| {
                Box::pin(async move {
                    let leader = MetaLeader::new(&meta_node);
                    leader.write(ent).await.map(|_| ())
                })
            })
            .await??;
    } else {
        // Forward to leader via gRPC
        let leader_node = meta_handle
            .handle_get_node(leader_id)
            .await?
            .ok_or_else(|| anyhow::anyhow!("leader node {} not found", leader_id))?;
        let leader_raft_endpoint = leader_node.endpoint;

        info!(
            "Forwarding register request to leader {} at {}",
            leader_id, leader_raft_endpoint
        );

        // Not the generated stub: a raft RPC that carries no shared secret is
        // refused by a leader running with `raft_secret_strict` on.
        let client = connect_raft_service(&leader_raft_endpoint, &config.raft_config).await?;

        let max_msg_size = config.raft_config.raft_grpc_max_message_size();
        let mut client = client
            .max_decoding_message_size(max_msg_size)
            .max_encoding_message_size(max_msg_size);

        let forward_req = ForwardRequest::new(1, ForwardRequestBody::Write(ent));
        let resp = client.forward(forward_req).await?;

View on GitHub (pinned to 288d84d76e)