{"record":{"id":"1de88956d5457cad","repo":"hashicorp/nomad","slug":"failed-to-remove-tracked-job-q-s-v","errorCode":null,"errorMessage":"failed to remove tracked job %q (%s): %v","messagePattern":"failed to remove tracked job %q \\((.+?)\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/periodic.go","lineNumber":268,"sourceCode":"\t})\n}\n\n// Remove stops tracking the passed job. If the job is not tracked, it is a\n// no-op. It assumes this is called while a lock is held.\nfunc (p *PeriodicDispatch) removeLocked(jobID structs.NamespacedID) error {\n\t// Do nothing if not enabled\n\tif !p.enabled {\n\t\treturn nil\n\t}\n\n\tjob, tracked := p.tracked[jobID]\n\tif !tracked {\n\t\treturn nil\n\t}\n\n\tdelete(p.tracked, jobID)\n\tif err := p.heap.Remove(job); err != nil {\n\t\treturn fmt.Errorf(\"failed to remove tracked job %q (%s): %v\", jobID.ID, jobID.Namespace, err)\n\t}\n\n\t// Signal an update.\n\tselect {\n\tcase p.updateCh <- struct{}{}:\n\tdefault:\n\t}\n\n\tp.logger.Debug(\"deregistered periodic job\", \"job\", job.NamespacedID())\n\treturn nil\n}\n\n// ForceEval causes the periodic job to be evaluated immediately and returns the\n// subsequent eval.\nfunc (p *PeriodicDispatch) ForceEval(namespace, jobID string) (*structs.Evaluation, error) {\n\tp.l.Lock()\n\n\t// Do nothing if not enabled","sourceCodeStart":250,"sourceCodeEnd":286,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/periodic.go#L250-L286","documentation":"removeLocked returns this when it deletes a job from the tracked map but the corresponding heap.Remove call fails. It means the heap and the tracked bookkeeping were out of sync for that job ID/namespace.","triggerScenarios":"PeriodicDispatch.Remove (or Add's removal branch via removeLocked) invoked for a tracked job whose entry is missing from the internal periodicHeap index — e.g. the heap was mutated elsewhere or state is inconsistent.","commonSituations":"Deregistering a periodic job during state restore or job deletion where the heap index lost the entry; concurrent modification bugs; leader transition edge cases.","solutions":["Check the wrapped error from heap.Remove to confirm the job truly isn't in the heap.","Verify removeLocked is only called while holding the dispatcher lock (p.l) so tracked and heap can't diverge.","Restart the Nomad server/leader election to rebuild the periodic dispatcher state.","Report a bug if tracked and heap diverge — this indicates an internal invariant violation."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Go: verify tracked+heap consistency before remove\n_, tracked := p.tracked[jobID]\nif !tracked { return nil }","typeGuard":null,"tryCatchPattern":"// Go\nif err := d.Remove(jobID); err != nil {\n    if strings.Contains(err.Error(), \"failed to remove tracked job\") {\n        log.Warn(\"heap/tracked divergence\", \"job\", jobID)\n    }\n}","preventionTips":["Always hold the dispatcher lock across remove operations","Treat already-removed jobs as idempotent success","Monitor for heap/tracked size mismatches after leader transitions"],"tags":["nomad","scheduler","periodic-jobs","state-inconsistency"],"backgroundTag":"inconsistent-internal-state","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}