hashicorp/nomad · error

UpsertScalingPolicies of policies failed: %v

Error message

UpsertScalingPolicies of policies failed: %v

What it means

Wraps an error from UpsertScalingPoliciesTxn when inserting the job's current scaling policies during job registration/update. Part of the same reconciliation that deletes stale policies; failure aborts the enclosing raft transaction.

Source

Thrown at nomad/state/state_store.go:5779

	deletedPolicies := []string{}
	iter, err := s.ScalingPoliciesByJobTxn(ws, job.Namespace, job.ID, txn)
	if err != nil {
		return fmt.Errorf("ScalingPoliciesByJob lookup failed: %v", err)
	}
	for raw := iter.Next(); raw != nil; raw = iter.Next() {
		oldPolicy := raw.(*structs.ScalingPolicy)
		if !newTargets[oldPolicy.JobKey()] {
			deletedPolicies = append(deletedPolicies, oldPolicy.ID)
		}
	}
	err = s.DeleteScalingPoliciesTxn(index, deletedPolicies, txn)
	if err != nil {
		return fmt.Errorf("DeleteScalingPolicies of removed policies failed: %v", err)
	}

	err = s.UpsertScalingPoliciesTxn(index, scalingPolicies, txn)
	if err != nil {
		return fmt.Errorf("UpsertScalingPolicies of policies failed: %v", err)
	}

	return nil
}

// updateJobSubmission stores the original job source and variables associated that the
// job structure originates from. It is up to the job submitter to include the source
// material, and as such sub may be nil, in which case nothing is stored.
func (s *StateStore) updateJobSubmission(index uint64, sub *structs.JobSubmission, namespace, jobID string, version uint64, txn *txn) error {
	// critical that we operate on a copy; the original must not be modified
	// e.g. in the case of job gc and its last second version bump
	sub = sub.Copy()

	switch {
	case sub == nil:
		return nil
	case namespace == "":
		return errors.New("job_submission requires a namespace")

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped error for the underlying cause
2. Validate scaling policy blocks in the job spec (type, target, min/max, policy fields)
3. Retry the job submission after correction

// validate scaling policy blocks before submitting
for _, tg := range job.TaskGroups {
    for _, sp := range tg.ScalingPolicies() {
        if sp.Min > sp.Max {
            return fmt.Errorf("scaling policy %s: min > max", sp.ID)
        }
        if sp.Type != "horizontal" {
            return fmt.Errorf("unsupported policy type %q", sp.Type)
        }
    }
}

if err != nil && strings.Contains(err.Error(), "UpsertScalingPolicies of policies failed") {
    return fmt.Errorf("job rejected: check scaling policy blocks: %w", err)
}

When it happens

Trigger: UpsertJob with scaling policy blocks present when UpsertScalingPoliciesTxn returns an insert error (memdb write failure, invalid policy schema in table).

Common situations: Submitting jobs with scaling blocks; user-facing cause could be a malformed scaling policy that fails deeper validation inside the upsert.

Related errors

• unable to update job scaling policies: %v
• getting job scaling policies for deletion failed: %v
• deleting scaling policy failed: %v
• ScalingPoliciesByJob lookup failed: %v
• DeleteScalingPolicies of removed policies failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8741a87b68856ad4. Report an issue: GitHub.