hashicorp/nomad · error

deleting job volume claims failed

Error message

deleting job volume claims failed: %v

What it means

The StateStore wraps errors from deleteTaskGroupHostVolumeClaimByNamespaceAndJob, which removes task-group host volume claims for the job being deleted. This wrapped error means the volume-claim cleanup step of job deletion failed inside the same memdb transaction, aborting the overall job deletion.

Solutions

  1. Read the wrapped inner error message for the real cause (it names the failing claim delete/lookup).
  2. Retry the job deregistration so the deletion runs in a fresh, valid transaction.
  3. Verify host-volume/CSI claim table consistency (nomad volume status) and remove orphaned claims if present.
  4. If persistent, check Raft store health and consider restoring state from a snapshot.

Example fix

// before
if err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {
	return fmt.Errorf("deleting job volume claims failed: %v", err)
}
// after (caller: retry deletion on a fresh txn)
err := s.StateStore.DeleteJob(index, namespace, jobID)
if err != nil {
	time.Sleep(backoff)
	err = s.StateStore.DeleteJob(index, namespace, jobID) // fresh txn
}
Defensive patterns

Strategy: retry

Validate before calling

// verify claims can be listed before deletion
_, err := state.hostVolumeClaimsByNamespaceJobTxn(nil, namespace, jobID)
if err != nil {
	return fmt.Errorf("cannot read claims for job %s/%s: %w", namespace, jobID, err)
}

Try / catch

err := state.DeleteJob(index, ns, jobID)
if err != nil && strings.Contains(err.Error(), "volume claims") {
	// inner cause is preserved; retry with a fresh transaction
	time.Sleep(backoff)
	err = state.DeleteJob(index, ns, jobID)
}

Prevention

When it happens

Trigger: Job deregistration (DeleteJob path) when the nested deleteTaskGroupHostVolumeClaimByNamespaceAndJob call fails — usually an inner memdb delete/lookup error (aborted txn, missing index table entry, or allocation failure) on the host volume claim tables.

Common situations: Clusters where jobs use host volumes with dynamic claims; frequently a downstream symptom of an earlier failed statement in the same delete transaction. Also hit when custom tooling writes directly to the state store and leaves it inconsistent.

Related errors


AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a3a295f76a8c278e. Report an issue: GitHub.

Appendix: source

Thrown at nomad/state/state_store.go:2119

	// Delete any remaining job scaling policies
	if err := s.deleteJobScalingPolicies(index, job, txn); err != nil {
		return fmt.Errorf("deleting job scaling policies failed: %v", err)
	}

	// Delete any job recommendations
	if err := s.deleteRecommendationsByJob(index, txn, job); err != nil {
		return fmt.Errorf("deleting job recommendatons failed: %v", err)
	}

	// Delete the scaling events
	if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
		return fmt.Errorf("deleting job scaling events failed: %v", err)
	}

	// Delete task group volume claims
	if err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {
		return fmt.Errorf("deleting job volume claims failed: %v", err)
	}

	if err := txn.Insert("index", &IndexEntry{"scaling_event", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// deleteJobScalingPolicies deletes any scaling policies associated with the job
func (s *StateStore) deleteJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {
	iter, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)
	if err != nil {
		return fmt.Errorf("getting job scaling policies for deletion failed: %v", err)
	}

	// Put them into a slice so there are no safety concerns while actually
	// performing the deletes

View on GitHub (pinned to 482b49bf1a)