hashicorp/nomad · error

failed to remove tracked job %q (%s): %v

Error message

failed to remove tracked job %q (%s): %v

What it means

removeLocked returns this when it deletes a job from the tracked map but the corresponding heap.Remove call fails. It means the heap and the tracked bookkeeping were out of sync for that job ID/namespace.

Source

Thrown at nomad/periodic.go:268

	})
}

// Remove stops tracking the passed job. If the job is not tracked, it is a
// no-op. It assumes this is called while a lock is held.
func (p *PeriodicDispatch) removeLocked(jobID structs.NamespacedID) error {
	// Do nothing if not enabled
	if !p.enabled {
		return nil
	}

	job, tracked := p.tracked[jobID]
	if !tracked {
		return nil
	}

	delete(p.tracked, jobID)
	if err := p.heap.Remove(job); err != nil {
		return fmt.Errorf("failed to remove tracked job %q (%s): %v", jobID.ID, jobID.Namespace, err)
	}

	// Signal an update.
	select {
	case p.updateCh <- struct{}{}:
	default:
	}

	p.logger.Debug("deregistered periodic job", "job", job.NamespacedID())
	return nil
}

// ForceEval causes the periodic job to be evaluated immediately and returns the
// subsequent eval.
func (p *PeriodicDispatch) ForceEval(namespace, jobID string) (*structs.Evaluation, error) {
	p.l.Lock()

	// Do nothing if not enabled

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped error from heap.Remove to confirm the job truly isn't in the heap.
2. Verify removeLocked is only called while holding the dispatcher lock (p.l) so tracked and heap can't diverge.
3. Restart the Nomad server/leader election to rebuild the periodic dispatcher state.
4. Report a bug if tracked and heap diverge — this indicates an internal invariant violation.

// Go: verify tracked+heap consistency before remove
_, tracked := p.tracked[jobID]
if !tracked { return nil }

// Go
if err := d.Remove(jobID); err != nil {
    if strings.Contains(err.Error(), "failed to remove tracked job") {
        log.Warn("heap/tracked divergence", "job", jobID)
    }
}

When it happens

Trigger: PeriodicDispatch.Remove (or Add's removal branch via removeLocked) invoked for a tracked job whose entry is missing from the internal periodicHeap index — e.g. the heap was mutated elsewhere or state is inconsistent.

Common situations: Deregistering a periodic job during state restore or job deletion where the heap index lost the entry; concurrent modification bugs; leader transition edge cases.

Related errors

• failed to add job %v: %v
• force launch for job %q failed: %v
• eval broker is enabled; eval broker must be paused to delete
• eval broker is enabled; eval broker must be paused to delete
• Job registration, dispatch, and scale are disabled by the sc

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1de88956d5457cad. Report an issue: GitHub.