hashicorp/nomad · error

all servers should be running version %v or later to use the

Error message

all servers should be running version %v or later to use the Nomad service provider

What it means

The Nomad native service registration provider requires every server in the local region to run at least minNomadServiceRegistrationVersion (Nomad 1.3.0), because the services RPC endpoints and state schema must exist everywhere. Upsert checks the peers cache via ServersMeetMinimumVersion and rejects the registration RPC if any server is older.

Source

Thrown at nomad/service_registration_endpoint.go:65

	}
	s.srv.MeasureRPCRate("service_registration", structs.RateMetricWrite, args)
	if err != nil {
		return structs.ErrPermissionDenied
	}
	defer metrics.MeasureSince([]string{"nomad", "service_registration", "upsert"}, time.Now())

	if err := auth.AuthorizeSameNodeServiceRegistrations(args.GetIdentity(), args.Services); err != nil {
		return err
	}

	// Nomad service registrations can only be used once all servers, in the
	// local region, have been upgraded to 1.3.0 or greater.
	if !s.srv.peersCache.ServersMeetMinimumVersion(
		s.srv.Region(),
		minNomadServiceRegistrationVersion,
		false,
	) {
		return fmt.Errorf(
			"all servers should be running version %v or later to use the Nomad service provider",
			minNomadServiceRegistrationVersion,
		)
	}

	// Use a multierror, so we can capture all validation errors and pass this
	// back so fixing in a single swoop.
	var mErr multierror.Error

	// Iterate the services and validate them. Any error results in the call
	// failing.
	for _, service := range args.Services {
		if err := service.Validate(); err != nil {
			mErr.Errors = append(mErr.Errors, err)
		}
	}
	if err := mErr.ErrorOrNil(); err != nil {
		return err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade all servers in the region to >= 1.3.0 (ideally the latest patch) before using the nomad service provider
2. Check nomad server members / nomad version on every server to find the lagging node
3. Temporarily use the consul provider for service registration until the upgrade completes
4. Verify servers finished upgrading (they appear in nomad server members with the new version, not draining/dead)

Example fix

// before (job uses nomad provider on 1.2.x cluster)
service { provider = "nomad" ... }
// after: upgrade all servers first
sudo nomad agent -server ... # version 1.3.0+ on every server
# then resubmit the job with provider = "nomad"

// Check every server in the region meets the minimum version before submitting nomad-provider services
const members = execSync('nomad server members -json').toString();
const min = '1.3.0';
const outdated = JSON.parse(members).filter(m => m.Status === 'alive' && cmpVersion(m.Version, min) < 0);
if (outdated.length) throw new Error('servers below ' + min + ': ' + outdated.map(m => m.Name).join(', '));

try {
  await nomad.services.register(serviceReg);
} catch (e) {
  if (String(e).includes('all servers should be running version')) {
    // fall back to consul provider or retry after cluster upgrade
    await registerWithConsul(serviceReg); // fallback
  } else throw e;
}

When it happens

Trigger: Calling the services registration RPC (JobRegister with nomad provider services, or direct Services.Upsert) while at least one server in the region runs a pre-1.3 Nomad binary.

Common situations: Mixed-version cluster during a rolling upgrade where jobs using provider = "nomad" are submitted early; agent that failed to upgrade and lags behind; multi-DC setup where a regional server is still old.

Related errors

• all servers should be running version %v or later to use JWT
• downgrading Raft is not supported, current version is %d, pr
• no default Consul services client
• unsupported minimum common raft protocol version
• service registration not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7d6ac1753035e6a8. Report an issue: GitHub.