hashicorp/nomad · error

periodicDispatcher.Remove failed: %w

Error message

periodicDispatcher.Remove failed: %w

What it means

During FSM application of a job deregistration, Nomad removes periodic jobs from the in-process periodic dispatcher. If periodicDispatcher.Remove returns an error (e.g. the job is not tracked as periodic or internal dispatcher state is inconsistent), the FSM wraps it with this message and aborts the Raft log entry application. The deregistration is not applied and the error is logged/returned to the caller.

Source

Thrown at nomad/fsm.go:875

	// store readers.
	return n.state.WithWriteTransaction(msgType, index, func(tx state.Txn) error {
		for jobNS, options := range req.Jobs {
			if err := n.handleJobDeregister(index, jobNS.ID, jobNS.Namespace, options.Purge, req.SubmitTime, false, tx); err != nil {
				n.logger.Error("deregistering job failed", "job", jobNS.ID, "error", err)
				return err
			}
		}
		return nil
	})
}

// handleJobDeregister is used to deregister a job. Leaves error logging up to
// caller.
func (n *nomadFSM) handleJobDeregister(index uint64, jobID, namespace string, purge bool, submitTime int64, noShutdownDelay bool, tx state.Txn) error {

	// If it is periodic remove it from the dispatcher
	if err := n.periodicDispatcher.Remove(namespace, jobID); err != nil {
		return fmt.Errorf("periodicDispatcher.Remove failed: %w", err)
	}

	if noShutdownDelay {
		ws := memdb.NewWatchSet()
		allocs, err := n.state.AllocsByJob(ws, namespace, jobID, false)
		if err != nil {
			return err
		}
		transition := &structs.DesiredTransition{NoShutdownDelay: new(true)}
		for _, alloc := range allocs {
			err := n.state.UpdateAllocDesiredTransitionTxn(tx, index, alloc.ID, transition)
			if err != nil {
				return err
			}
			err = tx.Insert("index", &state.IndexEntry{Key: "allocs", Value: index})
			if err != nil {
				return fmt.Errorf("index update failed: %v", err)
			}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped error: if it indicates the job was not tracked, verify whether the job is actually periodic via `nomad job inspect` and the periodic launch table.
2. Retry the job deregister operation; FSM apply is deterministic so a transient dispatcher inconsistency may resolve after leader election.
3. Restore from a consistent snapshot if state was restored from a backup from a different version/region.
4. If reproducible, file a Nomad issue with the wrapped error; Remove errors on tracked periodic jobs indicate internal state corruption.

Example fix

// client-side: retry deregister if it failed transiently
err = client.Jobs().Deregister(jobID, purge, nil)
if err != nil && strings.Contains(err.Error(), "periodicDispatcher.Remove failed") {
    err = client.Jobs().Deregister(jobID, purge, nil)
}

// verify the job is periodic before deregistering with purge
job, _, err := client.Jobs().Info(jobID, &nomad.QueryOptions{Namespace: ns})
if err != nil { return err }
if job.IsPeriodic() { /* expect dispatcher removal during deregister */ }

err := client.Jobs().Deregister(jobID, purge, nil)
if err != nil && strings.Contains(err.Error(), "periodicDispatcher.Remove failed") {
    // retry once after leader settles; else inspect state
}

When it happens

Trigger: apply job deregistration Raft entry for a job whose namespace/jobID is not registered in the periodic dispatcher, or dispatcher internal removal fails (hashset/key lookup error), during handleJobDeregister with purge or stop.

Common situations: Cluster state where the periodic launch table and dispatcher tracking diverge — e.g. after a partial restore/snapshot restore, job updated from periodic to non-periodic (or vice versa) across regions, or a bug in dispatcher bookkeeping.

Related errors

• failed adding job to periodic dispatcher: %v
• index update failed: %v
• DeleteJob failed: %w
• JobByID lookup failed: %w
• job %q in namespace %q doesn't exist to be deregistered

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c947d0fdf722e51d. Report an issue: GitHub.