hashicorp/nomad · error

deleting job volume claims failed: %v

Error message

deleting job volume claims failed: %v

What it means

The StateStore wraps errors from deleteTaskGroupHostVolumeClaimByNamespaceAndJob, which removes task-group host volume claims for the job being deleted. This wrapped error means the volume-claim cleanup step of job deletion failed inside the same memdb transaction, aborting the overall job deletion.

Source

Thrown at nomad/state/state_store.go:2119

	// Delete any remaining job scaling policies
	if err := s.deleteJobScalingPolicies(index, job, txn); err != nil {
		return fmt.Errorf("deleting job scaling policies failed: %v", err)
	}

	// Delete any job recommendations
	if err := s.deleteRecommendationsByJob(index, txn, job); err != nil {
		return fmt.Errorf("deleting job recommendatons failed: %v", err)
	}

	// Delete the scaling events
	if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
		return fmt.Errorf("deleting job scaling events failed: %v", err)
	}

	// Delete task group volume claims
	if err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {
		return fmt.Errorf("deleting job volume claims failed: %v", err)
	}

	if err := txn.Insert("index", &IndexEntry{"scaling_event", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// deleteJobScalingPolicies deletes any scaling policies associated with the job
func (s *StateStore) deleteJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {
	iter, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)
	if err != nil {
		return fmt.Errorf("getting job scaling policies for deletion failed: %v", err)
	}

	// Put them into a slice so there are no safety concerns while actually
	// performing the deletes

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped inner error message for the real cause (it names the failing claim delete/lookup).
2. Retry the job deregistration so the deletion runs in a fresh, valid transaction.
3. Verify host-volume/CSI claim table consistency (nomad volume status) and remove orphaned claims if present.
4. If persistent, check Raft store health and consider restoring state from a snapshot.

Example fix

// before
if err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {
	return fmt.Errorf("deleting job volume claims failed: %v", err)
}
// after (caller: retry deletion on a fresh txn)
err := s.StateStore.DeleteJob(index, namespace, jobID)
if err != nil {
	time.Sleep(backoff)
	err = s.StateStore.DeleteJob(index, namespace, jobID) // fresh txn
}

// verify claims can be listed before deletion
_, err := state.hostVolumeClaimsByNamespaceJobTxn(nil, namespace, jobID)
if err != nil {
	return fmt.Errorf("cannot read claims for job %s/%s: %w", namespace, jobID, err)
}

err := state.DeleteJob(index, ns, jobID)
if err != nil && strings.Contains(err.Error(), "volume claims") {
	// inner cause is preserved; retry with a fresh transaction
	time.Sleep(backoff)
	err = state.DeleteJob(index, ns, jobID)
}

When it happens

Trigger: Job deregistration (DeleteJob path) when the nested deleteTaskGroupHostVolumeClaimByNamespaceAndJob call fails — usually an inner memdb delete/lookup error (aborted txn, missing index table entry, or allocation failure) on the host volume claim tables.

Common situations: Clusters where jobs use host volumes with dynamic claims; frequently a downstream symptom of an earlier failed statement in the same delete transaction. Also hit when custom tooling writes directly to the state store and leaves it inconsistent.

Related errors

• task group volume claim does not exist
• deleting job scaling events failed: %v
• getting job scaling policies for deletion failed: %v
• deleting scaling policy failed: %v
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a3a295f76a8c278e. Report an issue: GitHub.