hashicorp/nomad · error

DeleteJob failed: %w

Error message

DeleteJob failed: %w

What it means

When a deregistration request has purge=true, handleJobDeregister deletes the job permanently via state.DeleteJobTxn. If the state-store delete fails (wrapped error), the FSM apply aborts. This means the job could not be removed from the persistent state within the transaction.

Source

Thrown at nomad/fsm.go:899

		if err != nil {
			return err
		}
		transition := &structs.DesiredTransition{NoShutdownDelay: new(true)}
		for _, alloc := range allocs {
			err := n.state.UpdateAllocDesiredTransitionTxn(tx, index, alloc.ID, transition)
			if err != nil {
				return err
			}
			err = tx.Insert("index", &state.IndexEntry{Key: "allocs", Value: index})
			if err != nil {
				return fmt.Errorf("index update failed: %v", err)
			}
		}
	}

	if purge {
		if err := n.state.DeleteJobTxn(index, namespace, jobID, tx); err != nil {
			return fmt.Errorf("DeleteJob failed: %w", err)
		}

		// We always delete from the periodic launch table because it is possible that
		// the job was updated to be non-periodic, thus checking if it is periodic
		// doesn't ensure we clean it up properly.
		n.state.DeletePeriodicLaunchTxn(index, namespace, jobID, tx)
		return nil
	}

	// Get the current job and mark it as stopped and re-insert it.
	ws := memdb.NewWatchSet()
	current, err := n.state.JobByIDTxn(ws, namespace, jobID, tx)
	if err != nil {
		return fmt.Errorf("JobByID lookup failed: %w", err)
	}

	if current == nil {
		return fmt.Errorf("job %q in namespace %q doesn't exist to be deregistered", jobID, namespace)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped error; if it is 'job not found', the job is already gone — no action needed.
2. Verify job existence with `nomad job status <jobID>` before purging.
3. Retry the purge after any in-flight deregistration completes.
4. Restore from a consistent snapshot if the state store appears corrupted.

Example fix

// avoid double-purge: check the job exists first
job, _, err := client.Jobs().Info(jobID, nil)
if err == nil {
    _, _, err = client.Jobs().Deregister(jobID, true, nil) // purge
}

// check the job still exists before purging
if _, _, err := client.Jobs().Info(jobID, &nomad.QueryOptions{Namespace: ns}); err != nil {
    // job already gone; skip purge
}

_, _, err := client.Jobs().Deregister(jobID, true, nil)
if err != nil && strings.Contains(err.Error(), "DeleteJob failed") {
    // check if 'not found' -> already purged; otherwise escalate
}

When it happens

Trigger: Job deregister with purge=true where DeleteJobTxn fails — e.g. job record missing in txn expectations, internal memdb error, or index constraint violation.

Common situations: Purging an already-purged job concurrently, corrupted state, or version-mismatched snapshot restore where expected records are absent.

Related errors

• failed adding job to periodic dispatcher: %v
• periodicDispatcher.Remove failed: %w
• index update failed: %v
• JobByID lookup failed: %w
• job %q in namespace %q doesn't exist to be deregistered

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fefdc017014e0693. Report an issue: GitHub.