hashicorp/nomad · error

unexpectedly did not forward Job.List to region %q

Error message

unexpectedly did not forward Job.List to region %q

What it means

namespaceTerminalJobsInRegion forwards a Job.List RPC to another region via the server's forwarding helper to check for remaining jobs during namespace deletion. The helper returns done=false when the RPC was not actually handled remotely; the code treats that as an internal invariant violation and returns this error naming the region. It indicates the forwarding layer failed to dispatch the request.

Source

Thrown at nomad/namespace_endpoint.go:331

	return true, nil
}

// namespaceTerminalJobsInRegion returns true if the namespace contains only
// terminal jobs in the given region.
func (n *Namespace) namespaceTerminalJobsInRegion(authToken, namespace, region string) (bool, error) {
	jobReq := &structs.JobListRequest{
		QueryOptions: structs.QueryOptions{
			Region:     region,
			Namespace:  namespace,
			AllowStale: false,
			AuthToken:  authToken,
		},
	}

	var jobResp structs.JobListResponse
	done, err := n.srv.forward("Job.List", jobReq, jobReq, &jobResp)
	if !done {
		return false, fmt.Errorf("unexpectedly did not forward Job.List to region %q", region)
	} else if err != nil {
		return false, err
	}

	for _, job := range jobResp.Jobs {
		if job.Status != structs.JobStatusDead {
			return false, nil
		}
	}
	return true, nil
}

// namespaceTerminalAllocsInRegion returns true if the namespace contains only
// terminal allocations in the given region.
func (n *Namespace) namespaceTerminalAllocsInRegion(authToken, namespace, region string) (bool, error) {
	allocReq := &structs.AllocListRequest{
		QueryOptions: structs.QueryOptions{
			Region:     region,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the target region exists and is healthy (nomad server members, cross-region gossip).
2. Check server logs for forwarding/RPC connection errors toward that region.
3. Correct region/federation configuration on both sides.
4. Retry the namespace deletion once federation is healthy.

// pre-check federation health before multi-region namespace delete
for _, region := range regions {
    if _, _, err := client.Agent().MembersOpts(&api.QueryOptions{Region: region}); err != nil {
        return fmt.Errorf("region %s unreachable: %w", region, err)
    }
}

err := deleteNamespacesWithRetry(client, names)
if err != nil && strings.Contains(err.Error(), "did not forward") {
    // do not hot-retry blindly: check region health/federation first
}

When it happens

Trigger: Deleting a namespace in a multi-region cluster where Job.List forwarding to a remote region fails to dispatch; remote region down or not federated; wrong region name in server config.

Common situations: Misconfigured region names; a peer region down or unreachable during namespace delete; partial federation where the region is advertised but not routable.

Related errors

• unexpectedly did not forward Alloc.List to region %q
• unexpectedly did not forward CSIVolume.List to region %q
• unexpectedly did not forward Variables.List to region %q
• No path to region
• controller list volumes: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ba10ec81729b0f9f. Report an issue: GitHub.