hashicorp/nomad · critical

Failed to start Raft: %v

Error message

Failed to start Raft: %v

What it means

If s.setupRaft() fails while NewServer is initializing consensus, the server shuts down and returns "Failed to start Raft: %v". setupRaft covers transport setup, log store creation, snapshot store creation, and raft bootstrap, so the embedded cause identifies the failing layer.

Source

Thrown at nomad/server.go:491

		s.logger.Error("failed to start RPC layer", "error", err)
		return nil, fmt.Errorf("Failed to start RPC layer: %v", err)
	}

	s.auth = auth.NewAuthenticator(&auth.AuthenticatorConfig{
		StateFn:        s.State,
		Logger:         s.logger,
		GetLeaderACLFn: s.getLeaderAcl,
		AclsEnabled:    s.config.ACLEnabled,
		VerifyTLS:      s.config.TLSConfig != nil && s.config.TLSConfig.EnableRPC && s.config.TLSConfig.VerifyServerHostname,
		Region:         s.Region(),
		Encrypter:      s.encrypter,
	})

	// Initialize the Raft server
	if err := s.setupRaft(); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start Raft", "error", err)
		return nil, fmt.Errorf("Failed to start Raft: %v", err)
	}

	// Initialize the wan Serf
	s.serf, err = s.setupSerf(config.SerfConfig, s.eventCh, serfSnapshot)
	if err != nil {
		s.Shutdown()
		s.logger.Error("failed to start serf WAN", "error", err)
		return nil, fmt.Errorf("Failed to start serf: %v", err)
	}

	// Initialize the scheduling workers
	if err := s.setupWorkers(s.shutdownCtx); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start workers", "error", err)
		return nil, fmt.Errorf("Failed to start workers: %v", err)
	}

	// Setup the Consul syncer

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped cause and fix accordingly (permissions, disk space, port conflict)
2. If raft state is corrupted, restore data_dir from backup or (accepting data loss) remove raft.db on a non-quorum node
3. Ensure each server has a unique node_id and never clone data_dir between servers
4. Verify data_dir exists and is writable by the nomad user

Example fix

// before
chown -R root:root /var/lib/nomad
// after
chown -R nomad:nomad /var/lib/nomad && systemctl restart nomad

// Pre-flight Raft prerequisites: writable data_dir and free raft port
if err := os.MkdirAll(cfg.DataDir, 0o700); err != nil {
    return fmt.Errorf("data_dir not writable: %w", err)
}
if f, err := os.Create(filepath.Join(cfg.DataDir, ".write-test")); err != nil {
    return err
} else { f.Close(); os.Remove(f.Name()) }

srv, err := nomad.NewServer(config, catalog, consulFn)
if err != nil {
    if strings.Contains(err.Error(), "Failed to start Raft") {
        // check data_dir permissions/disk and raft.db integrity before retry
    }
    return err
}

When it happens

Trigger: Starting a server where Raft initialization fails: unwritable data_dir (raft.db/snapshots), corrupted raft state, BoltDB log store open failure, transport bind failure on the raft port, or invalid bootstrap configuration.

Common situations: Disk full or permissions on data_dir preventing raft.db creation; corrupted BoltDB after a crash; raft port (4647) conflict; mismatched server IDs/state after cloning a VM image with existing data_dir.

Related errors

• unsupported minimum common raft protocol version
• No cluster leader
• server setup failed: %v
• Raft error when taking snapshot: %v
• No servers found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/04204e6f2a1b10fa. Report an issue: GitHub.