{"record":{"id":"ba10ec81729b0f9f","repo":"hashicorp/nomad","slug":"unexpectedly-did-not-forward-job-list-to-region-q","errorCode":null,"errorMessage":"unexpectedly did not forward Job.List to region %q","messagePattern":"unexpectedly did not forward Job\\.List to region %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/namespace_endpoint.go","lineNumber":331,"sourceCode":"\treturn true, nil\n}\n\n// namespaceTerminalJobsInRegion returns true if the namespace contains only\n// terminal jobs in the given region.\nfunc (n *Namespace) namespaceTerminalJobsInRegion(authToken, namespace, region string) (bool, error) {\n\tjobReq := &structs.JobListRequest{\n\t\tQueryOptions: structs.QueryOptions{\n\t\t\tRegion:     region,\n\t\t\tNamespace:  namespace,\n\t\t\tAllowStale: false,\n\t\t\tAuthToken:  authToken,\n\t\t},\n\t}\n\n\tvar jobResp structs.JobListResponse\n\tdone, err := n.srv.forward(\"Job.List\", jobReq, jobReq, &jobResp)\n\tif !done {\n\t\treturn false, fmt.Errorf(\"unexpectedly did not forward Job.List to region %q\", region)\n\t} else if err != nil {\n\t\treturn false, err\n\t}\n\n\tfor _, job := range jobResp.Jobs {\n\t\tif job.Status != structs.JobStatusDead {\n\t\t\treturn false, nil\n\t\t}\n\t}\n\treturn true, nil\n}\n\n// namespaceTerminalAllocsInRegion returns true if the namespace contains only\n// terminal allocations in the given region.\nfunc (n *Namespace) namespaceTerminalAllocsInRegion(authToken, namespace, region string) (bool, error) {\n\tallocReq := &structs.AllocListRequest{\n\t\tQueryOptions: structs.QueryOptions{\n\t\t\tRegion:     region,","sourceCodeStart":313,"sourceCodeEnd":349,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/namespace_endpoint.go#L313-L349","documentation":"namespaceTerminalJobsInRegion forwards a Job.List RPC to another region via the server's forwarding helper to check for remaining jobs during namespace deletion. The helper returns done=false when the RPC was not actually handled remotely; the code treats that as an internal invariant violation and returns this error naming the region. It indicates the forwarding layer failed to dispatch the request.","triggerScenarios":"Deleting a namespace in a multi-region cluster where Job.List forwarding to a remote region fails to dispatch; remote region down or not federated; wrong region name in server config.","commonSituations":"Misconfigured region names; a peer region down or unreachable during namespace delete; partial federation where the region is advertised but not routable.","solutions":["Verify the target region exists and is healthy (nomad server members, cross-region gossip).","Check server logs for forwarding/RPC connection errors toward that region.","Correct region/federation configuration on both sides.","Retry the namespace deletion once federation is healthy."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// pre-check federation health before multi-region namespace delete\nfor _, region := range regions {\n    if _, _, err := client.Agent().MembersOpts(&api.QueryOptions{Region: region}); err != nil {\n        return fmt.Errorf(\"region %s unreachable: %w\", region, err)\n    }\n}","typeGuard":null,"tryCatchPattern":"err := deleteNamespacesWithRetry(client, names)\nif err != nil && strings.Contains(err.Error(), \"did not forward\") {\n    // do not hot-retry blindly: check region health/federation first\n}","preventionTips":["Verify all peer regions are healthy and federated before cross-region deletes.","Monitor gossip/RPC connectivity between regions.","Correct region names in server/client config.","Back off and retry after restoring connectivity, not in a tight loop."],"tags":["nomad","rpc-forwarding","multi-region"],"backgroundTag":"rpc-forward-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}