hashicorp/nomad · error

deleting job scaling events failed: %v

Error message

deleting job scaling events failed: %v

What it means

Nomad's StateStore wraps every underlying memdb transaction error when deleting all scaling events for a job inside deleteJobScaling (part of the job-deletion transaction). The raw memdb error is preserved via %v so the operator can see the root cause (e.g. the transaction was already aborted or the table write failed). If this fires, the whole job deletion txn is rolled back.

Source

Thrown at nomad/state/state_store.go:2114

	// Delete the job submission
	if err := s.deleteJobSubmission(job, txn); err != nil {
		return fmt.Errorf("deleting job submission failed: %v", err)
	}

	// Delete any remaining job scaling policies
	if err := s.deleteJobScalingPolicies(index, job, txn); err != nil {
		return fmt.Errorf("deleting job scaling policies failed: %v", err)
	}

	// Delete any job recommendations
	if err := s.deleteRecommendationsByJob(index, txn, job); err != nil {
		return fmt.Errorf("deleting job recommendatons failed: %v", err)
	}

	// Delete the scaling events
	if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
		return fmt.Errorf("deleting job scaling events failed: %v", err)
	}

	// Delete task group volume claims
	if err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {
		return fmt.Errorf("deleting job volume claims failed: %v", err)
	}

	if err := txn.Insert("index", &IndexEntry{"scaling_event", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// deleteJobScalingPolicies deletes any scaling policies associated with the job
func (s *StateStore) deleteJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {
	iter, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)
	if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v cause in the log line — fix the root memdb/txn error it reports.
2. Retry the job deletion; memdb errors from a transient aborted txn resolve on a fresh transaction.
3. Check that no earlier error in the deleteJob path was swallowed, leaving the txn aborted before this DeleteAll.
4. If persistent, verify Raft/state store health (restore from backup or resync from a healthy peer).

Example fix

// before (caller ignoring wrapped cause)
if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
	return fmt.Errorf("deleting job scaling events failed: %v", err)
}
// after (caller-side: ensure txn is valid and surface cause)
if err := txn.Abort(); err != nil { /* txn already broken */ }
txn = s.db.Txn(true)
if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
	return fmt.Errorf("deleting job scaling events failed: %w", err)
}

// before calling job deregistration, verify job and txn state
if job == nil || job.ID == "" || job.Namespace == "" {
	return fmt.Errorf("job ID and namespace are required")
}
if txn == nil {
	return fmt.Errorf("job deletion requires a writable transaction")
}

func isTxnStateErr(err error) bool {
	return err != nil && strings.Contains(err.Error(), "transaction")
}

err := state.DeleteJob(index, namespace, jobID)
if err != nil {
	if strings.Contains(err.Error(), "deleting job scaling events") {
		// memdb txns are all-or-nothing: a fresh txn is safe to retry
		time.Sleep(backoff)
		err = state.DeleteJob(index, namespace, jobID)
	}
}

When it happens

Trigger: Calling job deregistration (Job.Unregister / state store DeleteJob) when the memdb txn.DeleteAll on table "scaling_event" fails — typically because the surrounding transaction was already in an error/aborted state (e.g. a prior statement in the same txn failed), memory pressure/allocation failure in memdb, or table corruption.

Common situations: Deleting jobs whose scaling events exist; usually surfaced as a cascade after an earlier failure in the same deleteJob transaction rather than a standalone problem. Also seen in tests or tooling that drive StateStore directly with a closed or aborted txn.

Related errors

• index update failed: %v
• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e1123b05a068eb04. Report an issue: GitHub.