hashicorp/nomad · error

getting job scaling policies for deletion failed: %v

Error message

getting job scaling policies for deletion failed: %v

What it means

deleteJobScalingPolicies first lists the job's scaling policies via ScalingPoliciesByJobTxn before deleting them. This wrapped error means that read/iteration failed, so the policy cleanup (and thus job deletion) aborts. It is a wrapped iterator/lookup error, not a delete error.

Source

Thrown at nomad/state/state_store.go:2133

	}

	// Delete task group volume claims
	if err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {
		return fmt.Errorf("deleting job volume claims failed: %v", err)
	}

	if err := txn.Insert("index", &IndexEntry{"scaling_event", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// deleteJobScalingPolicies deletes any scaling policies associated with the job
func (s *StateStore) deleteJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {
	iter, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)
	if err != nil {
		return fmt.Errorf("getting job scaling policies for deletion failed: %v", err)
	}

	// Put them into a slice so there are no safety concerns while actually
	// performing the deletes
	policies := []any{}
	for {
		raw := iter.Next()
		if raw == nil {
			break
		}
		policies = append(policies, raw)
	}

	// Do the deletes
	for _, p := range policies {
		if err := txn.Delete("scaling_policy", p); err != nil {
			return fmt.Errorf("deleting scaling policy failed: %v", err)
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped inner error to identify the failing memdb read.
2. Retry the job deletion with a fresh write transaction.
3. Ensure earlier statements in the same txn did not abort it before this call.
4. Check state store integrity (scaling_policy table) if the error persists across retries.

Example fix

// before
iter, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)
if err != nil {
	return fmt.Errorf("getting job scaling policies for deletion failed: %v", err)
}
// after (use %w so callers can errors.Is on the memdb cause)
if err != nil {
	return fmt.Errorf("getting job scaling policies for deletion failed: %w", err)
}

// verify the job exists before policy cleanup
job, err := state.JobByID(nil, namespace, jobID)
if err != nil || job == nil {
	return fmt.Errorf("job %s/%s not found or read failed: %w", namespace, jobID, err)
}

err := state.DeleteJob(index, ns, jobID)
if err != nil && strings.Contains(err.Error(), "getting job scaling policies") {
	time.Sleep(backoff)
	return state.DeleteJob(index, ns, jobID)
}

When it happens

Trigger: Job deregistration when ScalingPoliciesByJobTxn(nil, namespace, jobID, txn) errors — e.g. the underlying memdb iterator fails because the transaction is already aborted, or the "scaling_policy" table/index is unavailable or corrupted.

Common situations: Seen when an earlier step of the deleteJob transaction already failed (aborted txn reused), during snapshot/restore windows, or in tests driving the StateStore with malformed transactions.

Related errors

• deleting scaling policy failed: %v
• unable to update job scaling policies: %v
• deleting job scaling events failed: %v
• deleting job volume claims failed: %v
• ScalingPoliciesByJob lookup failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3a61e550055e2412. Report an issue: GitHub.