hashicorp/nomad · error
deleting job scaling events failed
Error message
deleting job scaling events failed: %v
What it means
Nomad's StateStore wraps every underlying memdb transaction error when deleting all scaling events for a job inside deleteJobScaling (part of the job-deletion transaction). The raw memdb error is preserved via %v so the operator can see the root cause (e.g. the transaction was already aborted or the table write failed). If this fires, the whole job deletion txn is rolled back.
Solutions
- Inspect the wrapped %v cause in the log line — fix the root memdb/txn error it reports.
- Retry the job deletion; memdb errors from a transient aborted txn resolve on a fresh transaction.
- Check that no earlier error in the deleteJob path was swallowed, leaving the txn aborted before this DeleteAll.
- If persistent, verify Raft/state store health (restore from backup or resync from a healthy peer).
Example fix
// before (caller ignoring wrapped cause)
if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
return fmt.Errorf("deleting job scaling events failed: %v", err)
}
// after (caller-side: ensure txn is valid and surface cause)
if err := txn.Abort(); err != nil { /* txn already broken */ }
txn = s.db.Txn(true)
if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
return fmt.Errorf("deleting job scaling events failed: %w", err)
} Defensive patterns
Strategy: try-catch
Validate before calling
// before calling job deregistration, verify job and txn state
if job == nil || job.ID == "" || job.Namespace == "" {
return fmt.Errorf("job ID and namespace are required")
}
if txn == nil {
return fmt.Errorf("job deletion requires a writable transaction")
} Type guard
func isTxnStateErr(err error) bool {
return err != nil && strings.Contains(err.Error(), "transaction")
} Try / catch
err := state.DeleteJob(index, namespace, jobID)
if err != nil {
if strings.Contains(err.Error(), "deleting job scaling events") {
// memdb txns are all-or-nothing: a fresh txn is safe to retry
time.Sleep(backoff)
err = state.DeleteJob(index, namespace, jobID)
}
} Prevention
- Always retry state store mutations on a fresh transaction — memdb txns are atomic and abort as a unit.
- Check the wrapped %v cause in logs before assuming the scaling_events table is the problem.
- Do not reuse a transaction after any prior statement returned an error.
- Monitor Raft/state store health so transient memdb failures are caught early.
When it happens
Trigger: Calling job deregistration (Job.Unregister / state store DeleteJob) when the memdb txn.DeleteAll on table "scaling_event" fails — typically because the surrounding transaction was already in an error/aborted state (e.g. a prior statement in the same txn failed), memory pressure/allocation failure in memdb, or table corruption.
Common situations: Deleting jobs whose scaling events exist; usually surfaced as a cascade after an earlier failure in the same deleteJob transaction rather than a standalone problem. Also seen in tests or tooling that drive StateStore directly with a closed or aborted txn.
Related errors
- ACL binding rule insert failed
- ACL binding rule lookup failed
- ACL policy lookup failed
- ACL role deletion failed
- ACL role insert failed
AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04).
Data as JSON: /api/errors/e1123b05a068eb04.
Report an issue: GitHub.
Appendix: source
Thrown at nomad/state/state_store.go:2114
// Delete the job submission
if err := s.deleteJobSubmission(job, txn); err != nil {
return fmt.Errorf("deleting job submission failed: %v", err)
}
// Delete any remaining job scaling policies
if err := s.deleteJobScalingPolicies(index, job, txn); err != nil {
return fmt.Errorf("deleting job scaling policies failed: %v", err)
}
// Delete any job recommendations
if err := s.deleteRecommendationsByJob(index, txn, job); err != nil {
return fmt.Errorf("deleting job recommendatons failed: %v", err)
}
// Delete the scaling events
if _, err = txn.DeleteAll("scaling_event", "id", namespace, jobID); err != nil {
return fmt.Errorf("deleting job scaling events failed: %v", err)
}
// Delete task group volume claims
if err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {
return fmt.Errorf("deleting job volume claims failed: %v", err)
}
if err := txn.Insert("index", &IndexEntry{"scaling_event", index}); err != nil {
return fmt.Errorf("index update failed: %v", err)
}
return nil
}
// deleteJobScalingPolicies deletes any scaling policies associated with the job
func (s *StateStore) deleteJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {
iter, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)
if err != nil {View on GitHub (pinned to 482b49bf1a)