hashicorp/nomad · error

allocation is terminal

Error message

allocation is terminal

What it means

verifyWorkloadIdentityClaim in nomad/auth/auth.go rejects workload-identity claims whose allocation is in a terminal client state (complete, failed, lost). Terminal allocations no longer run workloads, so signing/verifying new claims for them is treated as an expired claim. This prevents stale or re-played alloc claims from gaining credentials after the workload has stopped.

Source

Thrown at nomad/auth/auth.go:734

	return resolveAuthorizedClientNodePoolByNodeID(snap, aclObj, registration.NodeID)
}

func (s *Authenticator) verifyWorkloadIdentityClaim(claims *structs.IdentityClaims) error {
	snap, err := s.getState().Snapshot()
	if err != nil {
		return err
	}
	alloc, err := snap.AllocByID(nil, claims.AllocationID)
	if err != nil {
		return err
	}
	if alloc == nil || alloc.Job == nil {
		return fmt.Errorf("allocation does not exist")
	}

	// the claims for terminal allocs are always treated as expired
	if alloc.ClientTerminalStatus() {
		return fmt.Errorf("allocation is terminal")
	}

	return nil
}

func (s *Authenticator) resolveClaims(claims *structs.IdentityClaims) (*acl.ACL, error) {

	// Nomad node identity claims currently map to a client ACL. If we open this
	// up in the future, we will want to modify this section to perform similar
	// work that is done for workload claims.
	if claims.IsNode() {
		if claims.NodeIdentityClaims == nil || claims.NodeIdentityClaims.NodePool == "" {
			return nil, fmt.Errorf("node identity claims missing node pool")
		}
		return acl.NewClientACL(claims.NodeIdentityClaims.NodePool), nil
	}

	policies, err := s.ResolvePoliciesForClaims(claims)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the allocation's client status before verifying; if it is terminal (complete/failed/lost), do not request new credentials for it.
2. Re-run the workload so a fresh, non-terminal allocation is created and use its allocation ID/claims.
3. If the alloc looks terminal but should be running, check node health and reschedule; a 'lost' status usually means the node missed heartbeats.

Example fix

// before
claims, err := auth.VerifyClaim(ctx, req) // fails: "allocation is terminal"
// after
alloc, _ := client_allocs.GetAlloc(req.AllocationID)
if alloc != nil && !alloc.ClientTerminalStatus() {
    claims, err = auth.VerifyClaim(ctx, req)
} else {
    // reschedule or skip credential refresh for this alloc
}

alloc, _, err := client.Allocs().Info(allocID, nil)
if err != nil { return err }
if alloc.ClientStatus == "complete" || alloc.ClientStatus == "failed" || alloc.ClientStatus == "lost" {
    return fmt.Errorf("alloc %s is terminal; cannot verify claim", allocID)
}

When it happens

Trigger: Calling VerifyClaim (e.g. via the ACS/Workload Auth RPC) with a claim whose AllocationID resolves to an allocation whose ClientTerminalStatus() is true, i.e. the client reported the alloc as complete, failed, or lost.

Common situations: A client retries claim verification after the allocation finished; a lost node's allocs are marked lost by the server while the client still holds a valid-looking JWT; clock/retry issues cause verification long after job completion; drivers restart and re-authenticate dead allocs.

Related errors

• allocation does not exist
• no signed workload identity available
• no auth method config or client assertion
• missing AllocID
• missing login token

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fb001a2308376b94. Report an issue: GitHub.