neondatabase/neon · error

Migration to {node} rejected, may require `--force` ({})

Error message

Migration to {node} rejected, may require `--force` ({}) 

What it means

storcon_cli sent PUT control/v1/tenant/{tenant_shard_id}/migrate to the storage controller and got an HTTP 412 Precondition Failed ApiError back. The controller refuses the migration because the requested placement violates its preconditions (for example a node that is not a valid destination for this shard). The CLI rewraps the server message and hints at --force, which sets a flag in TenantShardMigrateRequest that bypasses those checks.

Source

Thrown at control_plane/storcon_cli/src/main.rs:731

                ..Default::default()
            };

            let req = TenantShardMigrateRequest {
                node_id: node,
                origin_node_id: None,
                migration_config,
            };

            match storcon_client
                .dispatch::<TenantShardMigrateRequest, TenantShardMigrateResponse>(
                    Method::PUT,
                    format!("control/v1/tenant/{tenant_shard_id}/migrate"),
                    Some(req),
                )
                .await
            {
                Err(mgmt_api::Error::ApiError(StatusCode::PRECONDITION_FAILED, msg)) => {
                    anyhow::bail!(
                        "Migration to {node} rejected, may require `--force` ({}) ",
                        msg
                    );
                }
                Err(e) => return Err(e.into()),
                Ok(_) => {}
            }

            watch_tenant_shard(storcon_client, tenant_shard_id, Some(node)).await?;
        }
        Command::TenantShardWatch { tenant_shard_id } => {
            watch_tenant_shard(storcon_client, tenant_shard_id, None).await?;
        }
        Command::TenantShardMigrateSecondary {
            tenant_shard_id,
            node,
        } => {
            let req = TenantShardMigrateRequest {

View on GitHub (pinned to 8f60b04da4)

Solutions

1. Add --force to the tenant-shard-migrate command if you intentionally want to override the controller's placement checks
2. Describe the tenant (control/v1/tenant/{tenant_id}) and check which node the shard is currently attached to and its secondaries, then request a placement that is actually a change
3. Verify the target node id exists, is Active, and does not conflict with the tenant's preferred AZ before retrying without --force

Example fix

# before
storcon_cli tenant-shard-migrate --tenant-shard-id <id> --node <node_id>
# after (when overriding is intended)
storcon_cli tenant-shard-migrate --tenant-shard-id <id> --node <node_id> --force

// Describe the tenant and inspect current placement before migrating
let desc: TenantDescribeResponse = client
    .dispatch(Method::GET, format!("control/v1/tenant/{tenant_id}"), None)
    .await?;
let shard = desc.shards.iter().find(|s| s.tenant_shard_id == *tenant_shard_id)
    .ok_or_else(|| anyhow::anyhow!("shard not in tenant"))?;
let target_change = shard.node_attached != Some(target_node)
    && !shard.node_secondary.contains(&target_node);
if !target_change && !force {
    anyhow::bail!("migration would be a no-op or violate preconditions; pass force");
}

match storcon_client.dispatch::<TenantShardMigrateRequest, TenantShardMigrateResponse>(
    Method::PUT,
    format!("control/v1/tenant/{tenant_shard_id}/migrate"),
    Some(req),
).await {
    Err(mgmt_api::Error::ApiError(StatusCode::PRECONDITION_FAILED, msg)) => {
        // decide: retry without change, or re-dispatch with force=true
        if cli.force {
            let forced = TenantShardMigrateRequest { migration_config: migration_config.force(), ..req };
            storcon_client.dispatch(Method::PUT, url, Some(forced)).await?;
        } else {
            return Err(anyhow::anyhow!("migration rejected by controller: {msg}"));
        }
    }
    Err(e) => return Err(e.into()),
    Ok(resp) => resp,
}

When it happens

Trigger: Calling storcon_cli tenant-shard-migrate without --force where the destination violates controller preconditions: migrating a shard to a node in the wrong role (e.g. the node is already the attached pageserver), a node excluded by the tenant's preferred AZ, an unavailable node, or otherwise rejected placement. The 412 response is matched explicitly in the dispatch error handler.

Common situations: scripting migrations without first describing the tenant's current attachment; forcing shards into an AZ that conflicts with the tenant's preferred AZ policy; node availability or scheduling changed between planning and executing the migration; operator intent really is to override the scheduler, so --force is the correct escape hatch.

Related errors

• Bulk migration requested away from node which doesn't exist.
• There are no nodes to migrate to
• Unknown scheduling policy '{s}', try active,essential,pause,
• Unknown availability state '{s}'
• AZ {} not found on any node: known AZs are: {:?}

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/5c7a06bc1e0c90b5. Report an issue: GitHub.