hashicorp/nomad · warning

failed to update job

Error message

failed to update job %q (%s) launch time: %v

What it means

When the job was already tracked, Add updates its position in the launch-time heap via p.heap.Update(job, next). If the heap update fails (job no longer present or heap invariant issues), the job ID, namespace, and error are wrapped and returned.

Solutions

1. Retry the job registration — the race is usually transient
2. Check for concurrent deregistration of the same periodic job in logs
3. Serialize job updates through the leader's state apply path (default) rather than direct dispatcher access
4. If it recurs at startup, inspect restorePeriodicDispatcher ordering and leader election churn

err := client.Jobs().Register(job, nil, nil)
if err != nil && strings.Contains(err.Error(), "failed to update job") {
    // transient heap race: brief wait and retry registration
    time.Sleep(time.Second); err = client.Jobs().Register(job, nil, nil)
}

When it happens

Trigger: p.heap.Update fails for an already-tracked job — typically a race where the job was removed from the heap concurrently (deregistration, periodic dispatch removal) between the tracked check and the Update call.

Common situations: Job deregistered by another request while an upsert for the same job is being applied; concurrent state-store applies racing the periodic dispatcher during leader handoff.

Related errors

• failed adding job
• heap doesn't contain object with ID
• can't force launch non-periodic job
• can't force run non-tracked job
• could not stat directory

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/53ac00179ff208dd. Report an issue: GitHub.

Appendix: source

Thrown at nomad/periodic.go:223

	_, tracked := p.tracked[tuple]
	if disabled {
		if tracked {
			p.removeLocked(tuple)
		}

		// If the job is disabled and we aren't tracking it, do nothing.
		return nil
	}

	// Add or update the job.
	p.tracked[tuple] = job
	next, err := job.Periodic.Next(time.Now().In(job.Periodic.GetLocation()))
	if err != nil {
		return fmt.Errorf("failed adding job %s: %v", job.NamespacedID(), err)
	}
	if tracked {
		if err := p.heap.Update(job, next); err != nil {
			return fmt.Errorf("failed to update job %q (%s) launch time: %v", job.ID, job.Namespace, err)
		}
		p.logger.Debug("updated periodic job", "job", job.NamespacedID())
	} else {
		if err := p.heap.Push(job, next); err != nil {
			return fmt.Errorf("failed to add job %v: %v", job.ID, err)
		}
		p.logger.Debug("registered periodic job", "job", job.NamespacedID())
	}

	// Signal an update.
	select {
	case p.updateCh <- struct{}{}:
	default:
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)