hashicorp/nomad · error

job delete failed: %v

Error message

job delete failed: %v

What it means

DeleteJobTxn (nomad/state/state_store.go:2020) wraps a failure from txn.Delete("jobs", existing) when removing the job row. The deletion failed inside the transaction, so the whole deregistration aborts and rolls back; nothing else (versions, deployments) is cleaned up.

Source

Thrown at nomad/state/state_store.go:2020

				if modified {
					// Update the modify index
					pSummary.ModifyIndex = index

					// Insert the summary
					if err := txn.Insert("job_summary", pSummary); err != nil {
						return fmt.Errorf("job summary insert failed: %v", err)
					}
					if err := txn.Insert("index", &IndexEntry{"job_summary", index}); err != nil {
						return fmt.Errorf("index update failed: %v", err)
					}
				}
			}
		}
	}

	// Delete the job
	if err := txn.Delete("jobs", existing); err != nil {
		return fmt.Errorf("job delete failed: %v", err)
	}
	if err := txn.Insert("index", &IndexEntry{"jobs", index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	// Delete the job versions
	if err := s.deleteJobVersions(index, job, txn); err != nil {
		return err
	}

	// Delete job deployments
	deployments, err := s.DeploymentsByJobID(nil, namespace, job.ID, true)
	if err != nil {
		return fmt.Errorf("deployment lookup for job %s failed: %v", job.ID, err)
	}

	deploymentIDs := []string{}
	for _, d := range deployments {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the deregistration; the txn rolled back so state is consistent
2. Check server logs for the wrapped %v root cause
3. Restart the Nomad server agent to rebuild state from Raft snapshots
4. Free resources / reduce load if errors correlate with server memory pressure

Example fix

// before
err := s.DeleteJobTxn(idx, ns, jobID, txn)
return err
// after
if err := s.DeleteJobTxn(idx, ns, jobID, txn); err != nil {
    if strings.Contains(err.Error(), "job delete failed") {
        return retryWithBackoff(func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })
    }
    return err
}

// confirm the job exists and is stoppable before deregister
job, err := stateStore.JobByID(nil, ns, jobID)
if err != nil { return err }
if job == nil { return nil } // avoids failed delete paths entirely

func isDeletable(j interface{}) bool { j2, ok := j.(*structs.Job); return ok && j2 != nil && j2.ID != "" }

if err := s.DeleteJobTxn(idx, ns, jobID, txn); err != nil {
    if strings.Contains(err.Error(), "job delete failed") {
        return retry(3, backoff, func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })
    }
    return err
}

When it happens

Trigger: Calling DeleteJob/DeleteJobTxn when txn.Delete on the jobs table errors for the existing job object — e.g. MemDB internal error or a write conflict within the txn.

Common situations: Server-side in-memory DB problems while applying Job.Deregister; corrupted job raw entries from a bad restore; resource exhaustion on heavily loaded servers.

Related errors

• error querying plugin %q: %v
• index update failed: %v
• JobByID lookup failed: %w
• UpsertJob failed: %w
• failed to restore from snapshot: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/913d03aca38b0bba. Report an issue: GitHub.