{"record":{"id":"1c02fb1162bbf430","repo":"hashicorp/nomad","slug":"region-q-v","errorCode":null,"errorMessage":"Region %q: %v","messagePattern":"Region %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"command/server_members.go","lineNumber":245,"sourceCode":"\t\t// Ignore left members\n\t\t// This prevents querying for leader status on regions where all members have left\n\t\tif m.Status == \"left\" {\n\t\t\tcontinue\n\t\t}\n\n\t\tregions[m.Tags[\"region\"]] = struct{}{}\n\t}\n\n\tif len(regions) == 0 {\n\t\treturn leaders, nil\n\t}\n\n\tvar mErr multierror.Error\n\tstatus := client.Status()\n\tfor reg := range regions {\n\t\tl, err := status.RegionLeader(reg)\n\t\tif err != nil {\n\t\t\t_ = multierror.Append(&mErr, fmt.Errorf(\"Region %q: %v\", reg, err))\n\t\t\tcontinue\n\t\t}\n\n\t\tleaders[reg] = l\n\t}\n\n\treturn leaders, mErr.ErrorOrNil()\n}\n\nfunc isLeader(member *api.AgentMember, leaders map[string]string) bool {\n\taddr := net.JoinHostPort(member.Addr, member.Tags[\"port\"])\n\treg := member.Tags[\"region\"]\n\tregLeader, ok := leaders[reg]\n\treturn ok && regLeader == addr\n}\n","sourceCodeStart":227,"sourceCodeEnd":261,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/server_members.go#L227-L261","documentation":"The `nomad server members` command queries leaders across all known regions. When `Status().RegionLeader(reg)` fails for a region (unreachable peers, RPC error, region down), the failure is accumulated into a multierror as `Region %q: %v` rather than aborting immediately, so the command can still report healthy regions' leaders.","triggerScenarios":"One or more federated regions are unreachable: RPC connection refused/timeout to a region's servers, the region's servers are all down, mTLS misconfiguration, or stale region membership data in the gossip/area config.","commonSituations":"Partial cluster outage where one datacenter/region's Nomad servers are offline; network partition or firewall blocking cross-region WAN federation ports (4646/4647); certificate expiry breaking cross-region mTLS; stale `server { enabled }` regions still listed by members.","solutions":["Inspect the wrapped %v error to identify which region and why (refused vs timeout vs TLS).","Check the failing region's server agents are running and reachable on RPC ports (4646/4647) across the WAN.","Verify mTLS certificates and CA trust between regions; renew expired certs.","Re-run `nomad server members` to see if the region recovered; remove decommissioned regions from federation if stale."],"exampleFix":"// before (firewall)\n# deny tcp 4647 between regions\n// after\n# allow tcp 4646,4647 between region servers\n# then verify:\nnomad server members -detailed","handlingStrategy":"retry","validationCode":"// Probe region reachability before collecting leaders\nstatus := client.Status()\nfor _, reg := range regions {\n    if _, err := status.RegionLeader(reg); err != nil {\n        log.Printf(\"region %s unreachable, will retry/skip: %v\", reg, err)\n    }\n}","typeGuard":null,"tryCatchPattern":"// Retry transient region RPC failures with backoff\nvar leader string\nerr := retry.Do(func() error {\n    l, err := status.RegionLeader(reg)\n    if err != nil {\n        return err\n    }\n    leader = l\n    return nil\n}, retry.Attempts(3), retry.Delay(2*time.Second))\nif err != nil {\n    _ = multierror.Append(&mErr, fmt.Errorf(\"Region %q: %v\", reg, err))\n}","preventionTips":["Monitor server agents in every federated region; alert on down nodes","Open firewall/RPC ports 4646-4647 across WAN for federation","Keep mTLS certificates between regions current and from a shared CA","Prune stale regions from federation and gossip before large status sweeps"],"tags":["nomad","network","rpc","federation"],"backgroundTag":"region-rpc-unreachable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}