What causes "AZ {} not found on any node: known AZs are: {:?}" in neondatabase/neon?

Re-run with one of the AZ ids listed in the error message itself (known AZs are printed)

AZ {} not found on any node: known AZs are: {:?}

Error message

AZ {} not found on any node: known AZs are: {:?}

What it means

While setting tenant preferred AZs, storcon_cli fetched the node list (GET control/v1/node), collected the set of availability_zone_id values, and the user-supplied preferred AZ is not in that set. The command aborts before sending the ShardsPreferredAzsRequest, so no state changes. The error prints the known AZs to make correction easy.

Source

Thrown at control_plane/storcon_cli/src/main.rs:905

                )
                .await?;

            // Learn about nodes to validate the AZ ID
            let nodes = storcon_client
                .dispatch::<(), Vec<NodeDescribeResponse>>(
                    Method::GET,
                    "control/v1/node".to_string(),
                    None,
                )
                .await?;

            if let Some(preferred_az) = &preferred_az {
                let azs = nodes
                    .into_iter()
                    .map(|n| (n.availability_zone_id))
                    .collect::<HashSet<_>>();
                if !azs.contains(preferred_az) {
                    anyhow::bail!(
                        "AZ {} not found on any node: known AZs are: {:?}",
                        preferred_az,
                        azs
                    );
                }
            } else {
                // Make it obvious to the user that since they've omitted an AZ, we're clearing it
                eprintln!("Clearing preferred AZ for tenant {tenant_id}");
            }

            // Construct a request that modifies all the tenant's shards
            let req = ShardsPreferredAzsRequest {
                preferred_az_ids: describe_response
                    .shards
                    .into_iter()
                    .map(|s| {
                        (
                            s.tenant_shard_id,

View on GitHub (pinned to 8f60b04da4)

Solutions

Re-run with one of the AZ ids listed in the error message itself (known AZs are printed)
List nodes (storcon_cli node list / GET control/v1/node) and copy the exact availability_zone_id string
If nodes lack the AZ you want, first configure the nodes' availability_zone_id, then retry the tenant command
Pass an empty AZ to deliberately clear the preference instead of guessing a placeholder name

Example fix

# before
storcon_cli tenant set-preferred-az --tenant-id <id> --az us-east-1a 
# after
storcon_cli tenant set-preferred-az --tenant-id <id> --az us-east-1a

Defensive patterns

Strategy: validation

Validate before calling

// Fetch known AZs and validate the requested one before calling the CLI/API
let nodes: Vec<NodeDescribeResponse> = client
    .dispatch(Method::GET, "control/v1/node".to_string(), None).await?;
let known_azs: HashSet<&str> = nodes.iter().map(|n| n.availability_zone_id.as_str()).collect();
if let Some(az) = &preferred_az {
    anyhow::ensure!(known_azs.contains(az.as_str()),
        "AZ '{az}' unknown; known AZs: {known_azs:?}");
}

Prevention

Derive AZ choices from node metadata at runtime instead of hardcoded strings
Trim and case-normalize AZ inputs before comparing
When clearing a preference, pass an explicitly empty value rather than a sentinel AZ name

When it happens

Trigger: Running the tenant-preferred-AZ command with an AZ string that does not exactly match any node's availability_zone_id: wrong casing, a zone name from a different deployment/region, or a typo. The HashSet membership check fails and the command bails.

Common situations: Copying AZ names from another environment or cloud account; nodes not yet configured with their AZ ids; trailing whitespace or case mismatch between the requested AZ and node metadata; assuming the flag accepts arbitrary AZ names that will be created later.

Related errors

AI-assisted analysis of neondatabase/neon@8f60b04da4 (2026-08-16). Data as JSON: /api/errors/309a515ae30572c1. Report an issue: GitHub.