hashicorp/nomad · error

Region %q: %v

Error message

Region %q: %v

What it means

The `nomad server members` command queries leaders across all known regions. When `Status().RegionLeader(reg)` fails for a region (unreachable peers, RPC error, region down), the failure is accumulated into a multierror as `Region %q: %v` rather than aborting immediately, so the command can still report healthy regions' leaders.

Source

Thrown at command/server_members.go:245

		// Ignore left members
		// This prevents querying for leader status on regions where all members have left
		if m.Status == "left" {
			continue
		}

		regions[m.Tags["region"]] = struct{}{}
	}

	if len(regions) == 0 {
		return leaders, nil
	}

	var mErr multierror.Error
	status := client.Status()
	for reg := range regions {
		l, err := status.RegionLeader(reg)
		if err != nil {
			_ = multierror.Append(&mErr, fmt.Errorf("Region %q: %v", reg, err))
			continue
		}

		leaders[reg] = l
	}

	return leaders, mErr.ErrorOrNil()
}

func isLeader(member *api.AgentMember, leaders map[string]string) bool {
	addr := net.JoinHostPort(member.Addr, member.Tags["port"])
	reg := member.Tags["region"]
	regLeader, ok := leaders[reg]
	return ok && regLeader == addr
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v error to identify which region and why (refused vs timeout vs TLS).
2. Check the failing region's server agents are running and reachable on RPC ports (4646/4647) across the WAN.
3. Verify mTLS certificates and CA trust between regions; renew expired certs.
4. Re-run `nomad server members` to see if the region recovered; remove decommissioned regions from federation if stale.

Example fix

// before (firewall)
# deny tcp 4647 between regions
// after
# allow tcp 4646,4647 between region servers
# then verify:
nomad server members -detailed

// Probe region reachability before collecting leaders
status := client.Status()
for _, reg := range regions {
    if _, err := status.RegionLeader(reg); err != nil {
        log.Printf("region %s unreachable, will retry/skip: %v", reg, err)
    }
}

// Retry transient region RPC failures with backoff
var leader string
err := retry.Do(func() error {
    l, err := status.RegionLeader(reg)
    if err != nil {
        return err
    }
    leader = l
    return nil
}, retry.Attempts(3), retry.Delay(2*time.Second))
if err != nil {
    _ = multierror.Append(&mErr, fmt.Errorf("Region %q: %v", reg, err))
}

When it happens

Trigger: One or more federated regions are unreachable: RPC connection refused/timeout to a region's servers, the region's servers are all down, mTLS misconfiguration, or stale region membership data in the gossip/area config.

Common situations: Partial cluster outage where one datacenter/region's Nomad servers are offline; network partition or firewall blocking cross-region WAN federation ports (4646/4647); certificate expiry breaking cross-region mTLS; stale `server { enabled }` regions still listed by members.

Related errors

• No path to node
• network namespace already exists but was misconfigured
• network already configured but not found in state
• no CNI network config found
• no servers

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1c02fb1162bbf430. Report an issue: GitHub.