{"record":{"id":"3b9e35f5e21b6a57","repo":"hashicorp/nomad","slug":"service-s-in-s-cannot-have-an-identity-until-all","errorCode":null,"errorMessage":"Service %s in %s cannot have an identity until all servers are upgraded to %s or later","messagePattern":"Service (.+?) in (.+?) cannot have an identity until all servers are upgraded to (.+?) or later","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/job_endpoint_hooks.go","lineNumber":567,"sourceCode":"\tif v.srv.config.JobMaxCount > 0 && totalCount > v.srv.config.JobMaxCount {\n\t\terr := fmt.Errorf(\"total count was greater than configured job_max_count: %d > %d\", totalCount, v.srv.config.JobMaxCount)\n\t\tmultierror.Append(validationErrors, err)\n\t}\n\n\treturn warnings, validationErrors.ErrorOrNil()\n}\n\nfunc (v *jobValidate) isEligibleForMultiIdentity() bool {\n\tif v.srv == nil || v.srv.serf == nil {\n\t\treturn true // handle tests w/o real servers safely\n\t}\n\treturn v.srv.peersCache.ServersMeetMinimumVersion(\n\t\tv.srv.Region(), minVersionMultiIdentities, true)\n}\n\nfunc (v *jobValidate) validateServiceIdentity(s *structs.Service, parent string, okForIdentity bool) error {\n\tif s.Identity != nil && !okForIdentity {\n\t\treturn fmt.Errorf(\"Service %s in %s cannot have an identity until all servers are upgraded to %s or later\",\n\t\t\ts.Name, parent, minVersionMultiIdentities)\n\t}\n\tif s.Identity != nil && s.Identity.Name == \"\" {\n\t\treturn fmt.Errorf(\"Service %s in %s has an identity with an empty name\", s.Name, parent)\n\t}\n\n\treturn nil\n}\n\n// validateVaultIdentity validates that a task is properly configured to access\n// a Vault cluster.\n//\n// It assumes the jobImplicitIdentitiesHook mutator hook has been called to\n// inject task identities if necessary.\nfunc (v *jobValidate) validateVaultIdentity(t *structs.Task, okForIdentity bool) ([]error, error) {\n\tvar warnings []error\n\n\tif t.Vault == nil {","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/job_endpoint_hooks.go#L549-L585","documentation":"Service identity blocks (service.identity) require all servers in the region to support multiple workload identities (minVersionMultiIdentities). If any server is older, declaring an identity on a service fails validation.","triggerScenarios":"Submitting a job where a service stanza has a non-nil identity field while the cluster has not fully upgraded to the required server version.","commonSituations":"Adopting workload-identity-based Consul integration on a mixed-version cluster; new job templates deployed before finishing a Nomad upgrade; canary server added but old servers still running.","solutions":["Complete server upgrades to the minimum multi-identity version","Remove the service identity block and use legacy Consul token/service auth until upgraded","Verify cluster readiness with the serversMeetMinimumVersion check (nomad version on all servers)"],"exampleFix":"// before\nservice { name = \"web\"; identity { name = \"consul-service-web\" } }\n// after\nservice { name = \"web\" }  # add identity after all servers upgraded","handlingStrategy":"validation","validationCode":"for _, tg := range job.TaskGroups {\n\tfor _, svc := range tg.Services {\n\t\tif svc.Identity != nil && !clusterSupportsMultiIdentity {\n\t\t\treturn fmt.Errorf(\"service %s identity requires upgraded servers\", svc.Name)\n\t\t}\n\t}\n}","typeGuard":"func hasServiceIdentities(tg *api.TaskGroup) bool {\n\treturn slices.ContainsFunc(tg.Services, func(s *api.Service) bool { return s.Identity != nil })\n}","tryCatchPattern":null,"preventionTips":["Finish rolling upgrades before adopting service identities","Feature-flag identity blocks per cluster version"],"tags":["nomad","workload-identity","consul","version-skew"],"backgroundTag":"server-version-skew","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}