hashicorp/nomad · error

error in job mutator %s: %v

Error message

error in job mutator %s: %v

What it means

Nomad runs job mutator hooks during job admission (Validate/registration). If any mutator's Mutate returns a non-nil error, admission aborts and this wrapped error, embedding the mutator's name and the underlying error, is returned to the caller.

Source

Thrown at nomad/job_endpoint_hooks.go:202

	}

	validateWarnings, err := j.admissionValidators(job)
	if err != nil {
		return nil, nil, err
	}
	warnings = append(warnings, validateWarnings...)

	return out, warnings, nil
}

// admissionMutator returns an updated job as well as warnings or an error.
func (j *Job) admissionMutators(job *structs.Job) (_ *structs.Job, warnings []error, err error) {
	var w []error
	for _, mutator := range j.mutators {
		job, w, err = mutator.Mutate(job)
		j.logger.Trace("job mutate results", "mutator", mutator.Name(), "warnings", w, "error", err)
		if err != nil {
			return nil, nil, fmt.Errorf("error in job mutator %s: %v", mutator.Name(), err)
		}
		warnings = append(warnings, w...)
	}
	return job, warnings, err
}

// admissionValidators returns a slice of validation warnings and a multierror
// of validation failures.
func (j *Job) admissionValidators(origJob *structs.Job) ([]error, error) {
	// ensure job is not mutated
	job := origJob.Copy()

	var warnings []error
	var errs error

	for _, validator := range j.validators {
		w, err := validator.Validate(job)
		j.logger.Trace("job validate results", "validator", validator.Name(), "warnings", w, "error", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the embedded mutator name (%s) and inner error to find which hook failed and why
2. Fix the job field the named mutator rejects (typically the connect/sidecar or interpolation block)
3. Upgrade Nomad if the failure is a known mutator bug
4. Temporarily remove the offending job stanza to confirm which hook errors

Example fix

// before
service { connect { sidecar_service { proxy { upstreams { destination_name = "" } } } } }
// after
service { connect { sidecar_service { proxy { upstreams { destination_name = "count-api" } } } } }

// run first: nomad job validate job.nomad.hcl (or POST /v1/validate) before register

if err := client.Jobs().Register(job, nil, nil); err != nil {
	var me *multierror.Error
	if errors.As(err, &me) {
		for _, e := range me.Errors { log.Printf("mutator failure: %v", e) }
	}
	return fmt.Errorf("job admission failed: %w", err)
}

When it happens

Trigger: Calling the Validate or job register API on a job where a registered mutator hook (e.g. the connect/interpolation mutator) fails while transforming the job.

Common situations: Malformed Consul Connect configuration that the connect interpolation mutator cannot process; custom/enterprise mutators choking on unsupported job fields; internal mutator bugs after version upgrades.

Related errors

• missing secret ID
• namespace cannot contain template delimiters or parenthesis
• wait config is nil or empty
• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d3c6e024a7967034. Report an issue: GitHub.