neondatabase/neon · error

Bulk migration requested away from node which doesn't exist.

Error message

Bulk migration requested away from node which doesn't exist.

What it means

In the bulk-migration (drain) flow, storcon_cli splits the node list into drain nodes and fill nodes by matching the requested node ids against control-plane node descriptors. If the number of requested ids does not equal the number of matched descriptors, some requested nodes do not exist, and the command bails before changing any scheduling state.

Source

Thrown at control_plane/storcon_cli/src/main.rs:1083

                    "control/v1/node".to_string(),
                    None,
                )
                .await?;

            let mut node_to_drain_descs = Vec::new();
            let mut node_to_fill_descs = Vec::new();

            for desc in node_descs {
                let to_drain = nodes.contains(&desc.id);
                if to_drain {
                    node_to_drain_descs.push(desc);
                } else {
                    node_to_fill_descs.push(desc);
                }
            }

            if nodes.len() != node_to_drain_descs.len() {
                anyhow::bail!("Bulk migration requested away from node which doesn't exist.")
            }

            node_to_fill_descs.retain(|desc| {
                matches!(desc.availability, NodeAvailabilityWrapper::Active)
                    && matches!(
                        desc.scheduling,
                        NodeSchedulingPolicy::Active | NodeSchedulingPolicy::Filling
                    )
            });

            if node_to_fill_descs.is_empty() {
                anyhow::bail!("There are no nodes to migrate to")
            }

            // Set the node scheduling policy to draining for the nodes which
            // we plan to drain.
            for node_desc in node_to_drain_descs.iter() {
                let req = NodeConfigureRequest {

View on GitHub (pinned to 8f60b04da4)

Solutions

1. List current nodes (storcon_cli node list / GET control/v1/node) and correct the node ids in the command
2. Remove ids of nodes that were already dropped or do not exist and re-run
3. Double-check for typos and for ids from a different storage-controller environment

Example fix

# before
storcon_cli tenant migrate-bulk --drain-nodes 1,2,99
# after (node 99 does not exist)
storcon_cli tenant migrate-bulk --drain-nodes 1,2

// Verify requested drain node ids exist before running bulk migration
let nodes_list: Vec<NodeDescribeResponse> = client
    .dispatch(Method::GET, "control/v1/node".to_string(), None).await?;
let existing: HashSet<NodeId> = nodes_list.iter().map(|n| n.id).collect();
let unknown: Vec<NodeId> = requested_drain_nodes.iter()
    .filter(|id| !existing.contains(*id)).cloned().collect();
anyhow::ensure!(unknown.is_empty(), "unknown node ids: {unknown:?}");

fn all_nodes_exist(requested: &[NodeId], existing: &HashSet<NodeId>) -> bool {
    requested.iter().all(|id| existing.contains(id))
}

When it happens

Trigger: Invoking the bulk migration command with a --drain-nodes-style list (or individual node args collected into `nodes`) where at least one node id is not present in GET control/v1/node results: a typo, a node that was already dropped, or a node id from a different environment.

Common situations: Copy-paste of stale node ids after nodes were replaced; scripts built from an old node inventory; typing node ids by hand; confusing node id with pageserver id or hostname.

Related errors

• Migration to {node} rejected, may require `--force` ({})
• AZ {} not found on any node: known AZs are: {:?}
• There are no nodes to migrate to
• Unknown scheduling policy '{s}', try active,essential,pause,
• Unknown availability state '{s}'

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/15860ce2488c8bbb. Report an issue: GitHub.