What causes "deleting scaling policy failed: %v" in hashicorp/nomad?

Read the wrapped cause for the specific failing policy.

deleting scaling policy failed: %v

Error message

deleting scaling policy failed: %v

What it means

After collecting a job's scaling policies into a slice, deleteJobScalingPolicies deletes each one with txn.Delete on the "scaling_policy" table. This wrapped error means one of those row deletions failed inside memdb, aborting the job-deletion transaction.

Source

Thrown at nomad/state/state_store.go:2150

	if err != nil {
		return fmt.Errorf("getting job scaling policies for deletion failed: %v", err)
	}

	// Put them into a slice so there are no safety concerns while actually
	// performing the deletes
	policies := []any{}
	for {
		raw := iter.Next()
		if raw == nil {
			break
		}
		policies = append(policies, raw)
	}

	// Do the deletes
	for _, p := range policies {
		if err := txn.Delete("scaling_policy", p); err != nil {
			return fmt.Errorf("deleting scaling policy failed: %v", err)
		}
	}

	if len(policies) > 0 {
		if err := txn.Insert("index", &IndexEntry{"scaling_policy", index}); err != nil {
			return fmt.Errorf("index update failed: %v", err)
		}
	}
	return nil
}

func (s *StateStore) deleteJobSubmission(job *structs.Job, txn *txn) error {
	// find submissions associated with job
	remove := *set.NewHashSet[*structs.JobSubmission, string](s.config.JobTrackedVersions)

	iter, err := txn.Get("job_submission", "id_prefix", job.Namespace, job.ID)
	if err != nil {
		return err

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the wrapped cause for the specific failing policy.
Retry the job deletion; the list-then-delete snapshot is rebuilt on retry.
Avoid concurrent policy writes against the same job while deregistering it.
If persistent, verify the scaling_policy table's integrity in the state store.

Example fix

// before
if err := txn.Delete("scaling_policy", p); err != nil {
	return fmt.Errorf("deleting scaling policy failed: %v", err)
}
// after (caller retrying with a fresh transaction)
for _, p := range policies {
	if err := txn.Delete("scaling_policy", p); err != nil {
		return fmt.Errorf("deleting scaling policy failed: %w", err)
	}
}

Defensive patterns

Strategy: retry

Validate before calling

// confirm policies are readable before deregistering
_, err := state.ScalingPoliciesByJob(nil, ns, jobID)
if err != nil {
	return err
}

Try / catch

err := state.DeleteJob(index, ns, jobID)
if err != nil && strings.Contains(err.Error(), "deleting scaling policy") {
	time.Sleep(backoff)
	return state.DeleteJob(index, ns, jobID)
}

Prevention

Avoid concurrent scaling-policy writes against a job being deleted.
Retry deregistration — list-then-delete is recomputed atomically per txn.
Surface the wrapped cause (%w) so callers can distinguish transient vs structural errors.
Check for orphaned scaling_policy rows after failed deletions.

When it happens

Trigger: Job deregistration when txn.Delete("scaling_policy", p) returns an error for one of the collected policies — memdb internal error, aborted transaction, or a row that vanished mid-transaction causing a delete mismatch.

Common situations: Concurrent modification of scaling policies while a job is being deleted; state store churn under load; tests that delete policies between the list and delete phases.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/abef02058254ccd3. Report an issue: GitHub.