{"record":{"id":"aac64cfa0929ba6d","repo":"databendlabs/databend","slug":"leader-node-not-found","errorCode":null,"errorMessage":"leader node {} not found","messagePattern":"leader node (.+?) not found","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/binaries/meta/entry.rs","lineNumber":264,"sourceCode":"    info!(\"Raft log entry for updating node: {:?}\", ent);\n\n    if meta_handle.id == leader_id {\n        // We are the leader: write directly via raft to avoid stale endpoint lookup.\n        info!(\"This node is the leader, writing directly\");\n        meta_handle\n            .request(move |meta_node| {\n                Box::pin(async move {\n                    let leader = MetaLeader::new(&meta_node);\n                    leader.write(ent).await.map(|_| ())\n                })\n            })\n            .await??;\n    } else {\n        // Forward to leader via gRPC\n        let leader_node = meta_handle\n            .handle_get_node(leader_id)\n            .await?\n            .ok_or_else(|| anyhow::anyhow!(\"leader node {} not found\", leader_id))?;\n        let leader_raft_endpoint = leader_node.endpoint;\n\n        info!(\n            \"Forwarding register request to leader {} at {}\",\n            leader_id, leader_raft_endpoint\n        );\n\n        // Not the generated stub: a raft RPC that carries no shared secret is\n        // refused by a leader running with `raft_secret_strict` on.\n        let client = connect_raft_service(&leader_raft_endpoint, &config.raft_config).await?;\n\n        let max_msg_size = config.raft_config.raft_grpc_max_message_size();\n        let mut client = client\n            .max_decoding_message_size(max_msg_size)\n            .max_encoding_message_size(max_msg_size);\n\n        let forward_req = ForwardRequest::new(1, ForwardRequestBody::Write(ent));\n        let resp = client.forward(forward_req).await?;","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/databendlabs/databend/blob/288d84d76e20a2f8f7173bda9691eb6ece301aa9/src/meta/binaries/meta/entry.rs#L246-L282","documentation":"In do_register, when the node being registered is not handled locally, the request is forwarded to the current raft leader over gRPC. The code first looks the leader up via meta_handle.handle_get_node(leader_id); if that returns None it raises this error because the leader's endpoint cannot be resolved. Without the endpoint the forwarding cannot proceed.","triggerScenarios":"register/leave requests arriving while cluster membership metadata is stale: leader_id from raft state refers to a node absent from the node registry (registry not yet synced after leader change, or the leader node was removed/de-registered).","commonSituations":"Right after a leader election where the new leader's node record hasn't propagated; operating on a cluster whose node list was manually edited; a joining node contacting an out-of-date follower.","solutions":["Retry registration after a short delay — membership usually converges and handle_get_node will then find the leader.","Verify cluster node list (admin API / nodes table) contains the node with leader_id and correct raft endpoint.","Check raft logs for recent leader changes or snapshot/compaction issues that could desynchronize the node registry.","If the leader record is genuinely gone, re-register/repair cluster membership (re-add the node) before forwarding registration."],"exampleFix":"// before\n.ok_or_else(|| anyhow::anyhow!(\"leader node {} not found\", leader_id))?;\n// after\nmatch meta_handle.handle_get_node(leader_id).await? {\n    Some(node) => node.endpoint,\n    None => return Err(anyhow::anyhow!(\n        \"leader node {} not found; cluster membership may be converging, retry\", leader_id)),\n}","handlingStrategy":"retry","validationCode":"// Rust: confirm the leader exists before forwarding\nlet known = meta_handle.handle_get_node(leader_id).await?.is_some();\nif !known { /* delay and re-read raft state before registering */ }","typeGuard":null,"tryCatchPattern":"let leader = retry(3, backoff, || async {\n    meta_handle.handle_get_node(leader_id).await?\n        .ok_or_else(|| anyhow!(\"leader {leader_id} not yet visible; retrying\"))\n}).await?;","preventionTips":["Retry registration with backoff around leader elections","Monitor cluster membership convergence after elections","Ensure all nodes read from the same raft snapshot before accepting registrations"],"tags":["rust","raft","meta-service","distributed"],"backgroundTag":"entity-not-found","analyzedSha":"288d84d76e20a2f8f7173bda9691eb6ece301aa9","analyzedAt":"2026-09-11T11:29:36.208Z","contentChangedAt":"2026-09-11T11:29:36.208Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}