{"record":{"id":"c947d0fdf722e51d","repo":"hashicorp/nomad","slug":"periodicdispatcher-remove-failed-w","errorCode":null,"errorMessage":"periodicDispatcher.Remove failed: %w","messagePattern":"periodicDispatcher\\.Remove failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/fsm.go","lineNumber":875,"sourceCode":"\t// store readers.\n\treturn n.state.WithWriteTransaction(msgType, index, func(tx state.Txn) error {\n\t\tfor jobNS, options := range req.Jobs {\n\t\t\tif err := n.handleJobDeregister(index, jobNS.ID, jobNS.Namespace, options.Purge, req.SubmitTime, false, tx); err != nil {\n\t\t\t\tn.logger.Error(\"deregistering job failed\", \"job\", jobNS.ID, \"error\", err)\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t\treturn nil\n\t})\n}\n\n// handleJobDeregister is used to deregister a job. Leaves error logging up to\n// caller.\nfunc (n *nomadFSM) handleJobDeregister(index uint64, jobID, namespace string, purge bool, submitTime int64, noShutdownDelay bool, tx state.Txn) error {\n\n\t// If it is periodic remove it from the dispatcher\n\tif err := n.periodicDispatcher.Remove(namespace, jobID); err != nil {\n\t\treturn fmt.Errorf(\"periodicDispatcher.Remove failed: %w\", err)\n\t}\n\n\tif noShutdownDelay {\n\t\tws := memdb.NewWatchSet()\n\t\tallocs, err := n.state.AllocsByJob(ws, namespace, jobID, false)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\ttransition := &structs.DesiredTransition{NoShutdownDelay: new(true)}\n\t\tfor _, alloc := range allocs {\n\t\t\terr := n.state.UpdateAllocDesiredTransitionTxn(tx, index, alloc.ID, transition)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\terr = tx.Insert(\"index\", &state.IndexEntry{Key: \"allocs\", Value: index})\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t\t}","sourceCodeStart":857,"sourceCodeEnd":893,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/fsm.go#L857-L893","documentation":"During FSM application of a job deregistration, Nomad removes periodic jobs from the in-process periodic dispatcher. If periodicDispatcher.Remove returns an error (e.g. the job is not tracked as periodic or internal dispatcher state is inconsistent), the FSM wraps it with this message and aborts the Raft log entry application. The deregistration is not applied and the error is logged/returned to the caller.","triggerScenarios":"apply job deregistration Raft entry for a job whose namespace/jobID is not registered in the periodic dispatcher, or dispatcher internal removal fails (hashset/key lookup error), during handleJobDeregister with purge or stop.","commonSituations":"Cluster state where the periodic launch table and dispatcher tracking diverge — e.g. after a partial restore/snapshot restore, job updated from periodic to non-periodic (or vice versa) across regions, or a bug in dispatcher bookkeeping.","solutions":["Check the wrapped error: if it indicates the job was not tracked, verify whether the job is actually periodic via `nomad job inspect` and the periodic launch table.","Retry the job deregister operation; FSM apply is deterministic so a transient dispatcher inconsistency may resolve after leader election.","Restore from a consistent snapshot if state was restored from a backup from a different version/region.","If reproducible, file a Nomad issue with the wrapped error; Remove errors on tracked periodic jobs indicate internal state corruption."],"exampleFix":"// client-side: retry deregister if it failed transiently\nerr = client.Jobs().Deregister(jobID, purge, nil)\nif err != nil && strings.Contains(err.Error(), \"periodicDispatcher.Remove failed\") {\n    err = client.Jobs().Deregister(jobID, purge, nil)\n}","handlingStrategy":"retry","validationCode":"// verify the job is periodic before deregistering with purge\njob, _, err := client.Jobs().Info(jobID, &nomad.QueryOptions{Namespace: ns})\nif err != nil { return err }\nif job.IsPeriodic() { /* expect dispatcher removal during deregister */ }","typeGuard":null,"tryCatchPattern":"err := client.Jobs().Deregister(jobID, purge, nil)\nif err != nil && strings.Contains(err.Error(), \"periodicDispatcher.Remove failed\") {\n    // retry once after leader settles; else inspect state\n}","preventionTips":["Avoid restoring snapshots from different Nomad versions or regions","Monitor server logs for periodic dispatcher warnings","Deregister periodic jobs via the standard job stop API, not ad-hoc state edits"],"tags":["nomad","raft","fsm","periodic-job"],"backgroundTag":"periodic-dispatcher-remove-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}