hashicorp/nomad · error

unexpectedly did not forward Job.List to region

Error message

unexpectedly did not forward Job.List to region %q

What it means

namespaceTerminalJobsInRegion forwards a Job.List RPC to another region via the server's forwarding helper to check for remaining jobs during namespace deletion. The helper returns done=false when the RPC was not actually handled remotely; the code treats that as an internal invariant violation and returns this error naming the region. It indicates the forwarding layer failed to dispatch the request.

Solutions

  1. Verify the target region exists and is healthy (nomad server members, cross-region gossip).
  2. Check server logs for forwarding/RPC connection errors toward that region.
  3. Correct region/federation configuration on both sides.
  4. Retry the namespace deletion once federation is healthy.
Defensive patterns

Strategy: retry

Validate before calling

// pre-check federation health before multi-region namespace delete
for _, region := range regions {
    if _, _, err := client.Agent().MembersOpts(&api.QueryOptions{Region: region}); err != nil {
        return fmt.Errorf("region %s unreachable: %w", region, err)
    }
}

Try / catch

err := deleteNamespacesWithRetry(client, names)
if err != nil && strings.Contains(err.Error(), "did not forward") {
    // do not hot-retry blindly: check region health/federation first
}

Prevention

When it happens

Trigger: Deleting a namespace in a multi-region cluster where Job.List forwarding to a remote region fails to dispatch; remote region down or not federated; wrong region name in server config.

Common situations: Misconfigured region names; a peer region down or unreachable during namespace delete; partial federation where the region is advertised but not routable.

Related errors


AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ba10ec81729b0f9f. Report an issue: GitHub.

Appendix: source

Thrown at nomad/namespace_endpoint.go:331

	return true, nil
}

// namespaceTerminalJobsInRegion returns true if the namespace contains only
// terminal jobs in the given region.
func (n *Namespace) namespaceTerminalJobsInRegion(authToken, namespace, region string) (bool, error) {
	jobReq := &structs.JobListRequest{
		QueryOptions: structs.QueryOptions{
			Region:     region,
			Namespace:  namespace,
			AllowStale: false,
			AuthToken:  authToken,
		},
	}

	var jobResp structs.JobListResponse
	done, err := n.srv.forward("Job.List", jobReq, jobReq, &jobResp)
	if !done {
		return false, fmt.Errorf("unexpectedly did not forward Job.List to region %q", region)
	} else if err != nil {
		return false, err
	}

	for _, job := range jobResp.Jobs {
		if job.Status != structs.JobStatusDead {
			return false, nil
		}
	}
	return true, nil
}

// namespaceTerminalAllocsInRegion returns true if the namespace contains only
// terminal allocations in the given region.
func (n *Namespace) namespaceTerminalAllocsInRegion(authToken, namespace, region string) (bool, error) {
	allocReq := &structs.AllocListRequest{
		QueryOptions: structs.QueryOptions{
			Region:     region,

View on GitHub (pinned to 482b49bf1a)