hashicorp/nomad · error

Service %s in %s cannot have an identity until all servers a

Error message

Service %s in %s cannot have an identity until all servers are upgraded to %s or later

What it means

Service identity blocks (service.identity) require all servers in the region to support multiple workload identities (minVersionMultiIdentities). If any server is older, declaring an identity on a service fails validation.

Source

Thrown at nomad/job_endpoint_hooks.go:567

	if v.srv.config.JobMaxCount > 0 && totalCount > v.srv.config.JobMaxCount {
		err := fmt.Errorf("total count was greater than configured job_max_count: %d > %d", totalCount, v.srv.config.JobMaxCount)
		multierror.Append(validationErrors, err)
	}

	return warnings, validationErrors.ErrorOrNil()
}

func (v *jobValidate) isEligibleForMultiIdentity() bool {
	if v.srv == nil || v.srv.serf == nil {
		return true // handle tests w/o real servers safely
	}
	return v.srv.peersCache.ServersMeetMinimumVersion(
		v.srv.Region(), minVersionMultiIdentities, true)
}

func (v *jobValidate) validateServiceIdentity(s *structs.Service, parent string, okForIdentity bool) error {
	if s.Identity != nil && !okForIdentity {
		return fmt.Errorf("Service %s in %s cannot have an identity until all servers are upgraded to %s or later",
			s.Name, parent, minVersionMultiIdentities)
	}
	if s.Identity != nil && s.Identity.Name == "" {
		return fmt.Errorf("Service %s in %s has an identity with an empty name", s.Name, parent)
	}

	return nil
}

// validateVaultIdentity validates that a task is properly configured to access
// a Vault cluster.
//
// It assumes the jobImplicitIdentitiesHook mutator hook has been called to
// inject task identities if necessary.
func (v *jobValidate) validateVaultIdentity(t *structs.Task, okForIdentity bool) ([]error, error) {
	var warnings []error

	if t.Vault == nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Complete server upgrades to the minimum multi-identity version
2. Remove the service identity block and use legacy Consul token/service auth until upgraded
3. Verify cluster readiness with the serversMeetMinimumVersion check (nomad version on all servers)

Example fix

// before
service { name = "web"; identity { name = "consul-service-web" } }
// after
service { name = "web" }  # add identity after all servers upgraded

for _, tg := range job.TaskGroups {
	for _, svc := range tg.Services {
		if svc.Identity != nil && !clusterSupportsMultiIdentity {
			return fmt.Errorf("service %s identity requires upgraded servers", svc.Name)
		}
	}
}

func hasServiceIdentities(tg *api.TaskGroup) bool {
	return slices.ContainsFunc(tg.Services, func(s *api.Service) bool { return s.Identity != nil })
}

When it happens

Trigger: Submitting a job where a service stanza has a non-nil identity field while the cluster has not fully upgraded to the required server version.

Common situations: Adopting workload-identity-based Consul integration on a mixed-version cluster; new job templates deployed before finishing a Nomad upgrade; canary server added but old servers still running.

Related errors

• error getting signed identity for task %s: %v
• error getting signed identity for service %s: %v
• tasks can only have 1 identity block until all servers are u
• Service %s in %s has an identity with an empty name
• error creating bootstrap configuration for Connect proxy sid

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3b9e35f5e21b6a57. Report an issue: GitHub.