{"record":{"id":"7d6ac1753035e6a8","repo":"hashicorp/nomad","slug":"all-servers-should-be-running-version-v-or-later-7d6ac1","errorCode":null,"errorMessage":"all servers should be running version %v or later to use the Nomad service provider","messagePattern":"all servers should be running version (.+?) or later to use the Nomad service provider","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/service_registration_endpoint.go","lineNumber":65,"sourceCode":"\t}\n\ts.srv.MeasureRPCRate(\"service_registration\", structs.RateMetricWrite, args)\n\tif err != nil {\n\t\treturn structs.ErrPermissionDenied\n\t}\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"service_registration\", \"upsert\"}, time.Now())\n\n\tif err := auth.AuthorizeSameNodeServiceRegistrations(args.GetIdentity(), args.Services); err != nil {\n\t\treturn err\n\t}\n\n\t// Nomad service registrations can only be used once all servers, in the\n\t// local region, have been upgraded to 1.3.0 or greater.\n\tif !s.srv.peersCache.ServersMeetMinimumVersion(\n\t\ts.srv.Region(),\n\t\tminNomadServiceRegistrationVersion,\n\t\tfalse,\n\t) {\n\t\treturn fmt.Errorf(\n\t\t\t\"all servers should be running version %v or later to use the Nomad service provider\",\n\t\t\tminNomadServiceRegistrationVersion,\n\t\t)\n\t}\n\n\t// Use a multierror, so we can capture all validation errors and pass this\n\t// back so fixing in a single swoop.\n\tvar mErr multierror.Error\n\n\t// Iterate the services and validate them. Any error results in the call\n\t// failing.\n\tfor _, service := range args.Services {\n\t\tif err := service.Validate(); err != nil {\n\t\t\tmErr.Errors = append(mErr.Errors, err)\n\t\t}\n\t}\n\tif err := mErr.ErrorOrNil(); err != nil {\n\t\treturn err","sourceCodeStart":47,"sourceCodeEnd":83,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/service_registration_endpoint.go#L47-L83","documentation":"The Nomad native service registration provider requires every server in the local region to run at least minNomadServiceRegistrationVersion (Nomad 1.3.0), because the services RPC endpoints and state schema must exist everywhere. Upsert checks the peers cache via ServersMeetMinimumVersion and rejects the registration RPC if any server is older.","triggerScenarios":"Calling the services registration RPC (JobRegister with nomad provider services, or direct Services.Upsert) while at least one server in the region runs a pre-1.3 Nomad binary.","commonSituations":"Mixed-version cluster during a rolling upgrade where jobs using provider = \"nomad\" are submitted early; agent that failed to upgrade and lags behind; multi-DC setup where a regional server is still old.","solutions":["Upgrade all servers in the region to >= 1.3.0 (ideally the latest patch) before using the nomad service provider","Check nomad server members / nomad version on every server to find the lagging node","Temporarily use the consul provider for service registration until the upgrade completes","Verify servers finished upgrading (they appear in nomad server members with the new version, not draining/dead)"],"exampleFix":"// before (job uses nomad provider on 1.2.x cluster)\nservice { provider = \"nomad\" ... }\n// after: upgrade all servers first\nsudo nomad agent -server ... # version 1.3.0+ on every server\n# then resubmit the job with provider = \"nomad\"","handlingStrategy":"validation","validationCode":"// Check every server in the region meets the minimum version before submitting nomad-provider services\nconst members = execSync('nomad server members -json').toString();\nconst min = '1.3.0';\nconst outdated = JSON.parse(members).filter(m => m.Status === 'alive' && cmpVersion(m.Version, min) < 0);\nif (outdated.length) throw new Error('servers below ' + min + ': ' + outdated.map(m => m.Name).join(', '));","typeGuard":null,"tryCatchPattern":"try {\n  await nomad.services.register(serviceReg);\n} catch (e) {\n  if (String(e).includes('all servers should be running version')) {\n    // fall back to consul provider or retry after cluster upgrade\n    await registerWithConsul(serviceReg); // fallback\n  } else throw e;\n}","preventionTips":["Complete server rolling upgrades before submitting jobs that use provider = \"nomad\"","Monitor nomad server members versions continuously; alert on skew","Gate job specs using the nomad provider on cluster version in your deploy pipeline"],"tags":["service-registration","versioning","cluster","upgrade"],"backgroundTag":"servers-below-minimum-version","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}