hashicorp/nomad · warning

%s %d: job exists with conflicting job modify index: %d

Error message

%s %d: job exists with conflicting job modify index: %d

What it means

The Job.Scale RPC supports optimistic concurrency: if the caller passes JobModifyIndex, Nomad verifies it still matches the job's current JobModifyIndex before applying the scale. A mismatch means the job was modified (registered, scaled, etc.) since the caller read it, so the scale is rejected with the RegisterEnforceIndexErrPrefix to force a re-read.

Source

Thrown at nomad/job_endpoint.go:1070

		// Update group count
		group.Count = newCount
		job.SubmitTime = now

		// Block scaling event if there's an active deployment
		deployment, err := snap.LatestDeploymentByJobID(ws, namespace, args.JobID)
		if err != nil {
			j.logger.Error("unable to lookup latest deployment", "error", err)
			return err
		}

		if deployment != nil && deployment.Active() && deployment.JobCreateIndex == job.CreateIndex {
			return structs.NewErrRPCCoded(400, "job scaling blocked due to active deployment")
		}

		// If JobModifyIndex set, check it before trying to apply
		if args.JobModifyIndex > 0 {
			if args.JobModifyIndex != job.JobModifyIndex {
				return fmt.Errorf("%s %d: job exists with conflicting job modify index: %d",
					structs.RegisterEnforceIndexErrPrefix, args.JobModifyIndex, job.JobModifyIndex)
			}
		}

		// Commit the job update
		_, jobModifyIndex, err := j.srv.raftApply(
			structs.JobRegisterRequestType,
			structs.JobRegisterRequest{
				Job:            job,
				EnforceIndex:   true,
				JobModifyIndex: job.JobModifyIndex,
				PolicyOverride: args.PolicyOverride,
				WriteRequest:   args.WriteRequest,
			},
		)
		if err != nil {
			j.logger.Error("job register for scale failed", "error", err)
			return err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Re-fetch the current job (GET /v1/job/<id>), take its fresh JobModifyIndex, and retry the scale
2. Retry with backoff on conflict since concurrent writers resolve quickly
3. Pass JobModifyIndex: 0 to skip the check only if last-write-wins is acceptable (risky)
4. Serialize scaling for the job through a single controller/lock

Example fix

// before
job, _, _ := client.Jobs().Info("web", nil)
time.Sleep(5 * time.Minute)
client.Jobs().Scale("web", "web", "group", &job.JobModifyIndex, ...)
// after
job, _, _ := client.Jobs().Info("web", nil)
_, _, err := client.Jobs().Scale("web", "web", "group", job.JobModifyIndex, ...)
if err != nil {
    job, _, _ = client.Jobs().Info("web", nil) // refresh index and retry
}

job, _, _ := client.Jobs().Info(jobID, nil)
modifyIndex := *job.JobModifyIndex // read immediately before scaling

for i := 0; i < 3; i++ {
	job, _, _ = client.Jobs().Info(jobID, nil)
	_, _, err := client.Jobs().Scale(jobID, jobID, group, job.JobModifyIndex, uint64(count), "api", false, nil)
	if err == nil { break }
	if !strings.Contains(err.Error(), "conflicting job modify index") { return err }
	time.Sleep(time.Duration(1<<i) * time.Second)
}

When it happens

Trigger: POST /v1/job/<id>/scale with args.JobModifyIndex > 0 that differs from the live job.JobModifyIndex — concurrent scaling/deploys, a stale job object cached from an earlier read, or an autoscaler racing a human 'nomad job run'.

Common situations: Autoscaler and CI pipeline scaling the same job simultaneously; long-lived scripts using an index captured minutes earlier; clients copying JobModifyIndex from a different job's info response.

Related errors

• ErrLockConflict
• root key in use, cannot delete
• tasks %s and %s want to initiate networking but only one dri
• Group %v not found within job
• Current job has version %d; enforcing version %d

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/14437bb2288df509. Report an issue: GitHub.