hashicorp/nomad · error

ScalingPoliciesByJob lookup failed: %v

Error message

ScalingPoliciesByJob lookup failed: %v

What it means

Wraps an error from ScalingPoliciesByJobTxn while reconciling scaling policies during job update. It means the memdb iterator over scaling policies for the job's namespace/ID failed. Without this lookup Nomad cannot determine which existing policies to delete.

Source

Thrown at nomad/state/state_store.go:5764

	return nil
}

// updateJobScalingPolicies upserts any scaling policies contained in the job and removes
// any previous scaling policies that were removed from the job
func (s *StateStore) updateJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {

	ws := memdb.NewWatchSet()

	scalingPolicies := job.GetScalingPolicies()
	newTargets := map[string]bool{}
	for _, p := range scalingPolicies {
		newTargets[p.JobKey()] = true
	}
	// find existing policies that need to be deleted
	deletedPolicies := []string{}
	iter, err := s.ScalingPoliciesByJobTxn(ws, job.Namespace, job.ID, txn)
	if err != nil {
		return fmt.Errorf("ScalingPoliciesByJob lookup failed: %v", err)
	}
	for raw := iter.Next(); raw != nil; raw = iter.Next() {
		oldPolicy := raw.(*structs.ScalingPolicy)
		if !newTargets[oldPolicy.JobKey()] {
			deletedPolicies = append(deletedPolicies, oldPolicy.ID)
		}
	}
	err = s.DeleteScalingPoliciesTxn(index, deletedPolicies, txn)
	if err != nil {
		return fmt.Errorf("DeleteScalingPolicies of removed policies failed: %v", err)
	}

	err = s.UpsertScalingPoliciesTxn(index, scalingPolicies, txn)
	if err != nil {
		return fmt.Errorf("UpsertScalingPolicies of policies failed: %v", err)
	}

	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped inner error for the memdb cause
2. Retry the job submission; the raft txn aborted atomically
3. Verify scaling policy table health via Nomad operator debug/endpoint if recurring

if err != nil && strings.Contains(err.Error(), "ScalingPoliciesByJob lookup failed") {
    // nothing persisted; resubmit job
    return retryWithBackoff(func() error { _, _, err := jobs.Register(job, nil); return err })
}

When it happens

Trigger: UpsertJob (job update with scaling policy reconciliation) when ScalingPoliciesByJobTxn returns a memdb lookup/iteration error for the job's namespace and ID.

Common situations: Internal memdb read failure during job submission with scaling policy blocks; rare, tied to state store health.

Related errors

• unable to update job scaling policies: %v
• getting job scaling policies for deletion failed: %v
• deleting scaling policy failed: %v
• DeleteScalingPolicies of removed policies failed: %v
• UpsertScalingPolicies of policies failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ef3bd9e281b78b3b. Report an issue: GitHub.