hashicorp/nomad · error

Failed to start workers: %v

Error message

Failed to start workers: %v

What it means

After Raft and Serf come up, NewServer starts the scheduling workers via setupWorkers(s.shutdownCtx). Any failure there (worker/eval broker initialization) shuts the server down and returns "Failed to start workers: %v". This is rare and usually reflects an internal configuration or resource problem rather than external state.

Source

Thrown at nomad/server.go:506

	if err := s.setupRaft(); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start Raft", "error", err)
		return nil, fmt.Errorf("Failed to start Raft: %v", err)
	}

	// Initialize the wan Serf
	s.serf, err = s.setupSerf(config.SerfConfig, s.eventCh, serfSnapshot)
	if err != nil {
		s.Shutdown()
		s.logger.Error("failed to start serf WAN", "error", err)
		return nil, fmt.Errorf("Failed to start serf: %v", err)
	}

	// Initialize the scheduling workers
	if err := s.setupWorkers(s.shutdownCtx); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start workers", "error", err)
		return nil, fmt.Errorf("Failed to start workers: %v", err)
	}

	// Setup the Consul syncer
	if err := s.setupConsulSyncer(); err != nil {
		s.logger.Error("failed to create server consul syncer", "error", err)
		return nil, fmt.Errorf("failed to create server Consul syncer: %v", err)
	}

	// Setup the deployment watcher.
	if err := s.setupDeploymentWatcher(); err != nil {
		s.logger.Error("failed to create deployment watcher", "error", err)
		return nil, fmt.Errorf("failed to create deployment watcher: %v", err)
	}

	// Setup the volume watcher
	if err := s.setupVolumeWatcher(); err != nil {
		s.logger.Error("failed to create volume watcher", "error", err)
		return nil, fmt.Errorf("failed to create volume watcher: %v", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped cause from the server log ("failed to start workers") and fix the named configuration problem
2. Remove unusual scheduler/worker overrides in the server config and retry with defaults
3. Confirm the binary matches the intended edition (OSS vs Enterprise) and version consistency across config
4. If persistent, file an issue with the underlying error since this indicates an unexpected internal failure

Example fix

// before
server {
  default_scheduler_config = "bogus"
}
// after
server { enabled = true } # worker/scheduler settings at defaults

// Validate server config defaults before startup in embedders
if cfg.DefaultSchedulerConfig != (structs.SchedulerConfiguration{}) {
    // ensure values are ones your build supports
}

srv, err := nomad.NewServer(config, catalog, consulFn)
if err != nil {
    if strings.Contains(err.Error(), "Failed to start workers") {
        log.Printf("worker init failed, retry with default server config: %v", err)
    }
    return err
}

When it happens

Trigger: Starting a server where setupWorkers fails — e.g. invalid worker-related server configuration producing a bad scheduler config, or context/resource initialization errors while creating the eval broker and worker goroutines.

Common situations: Custom builds or enterprise/oss scheduler mismatches; extremely constrained environments; misconfigured server config that reaches worker setup with invalid values; regression after upgrading with modified defaults.

Related errors

• timeout cannot be negative
• eval broker disabled
• Evaluation ID not found
• Token does not match for Evaluation ID
• Evaluation ID Ack'd after Nack timer expiration

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/057252dfa8b61bce. Report an issue: GitHub.