hashicorp/nomad · critical

Failed to start serf: %v

Error message

Failed to start serf: %v

What it means

NewServer wraps failures from s.setupSerf for the WAN Serf membership layer with "Failed to start serf: %v" after calling Shutdown. Serf setup creates the membership and event handlers and binds its gossip port, so failures are usually bind conflicts, bad advertise addresses, or a corrupt Serf snapshot.

Source

Thrown at nomad/server.go:499

		AclsEnabled:    s.config.ACLEnabled,
		VerifyTLS:      s.config.TLSConfig != nil && s.config.TLSConfig.EnableRPC && s.config.TLSConfig.VerifyServerHostname,
		Region:         s.Region(),
		Encrypter:      s.encrypter,
	})

	// Initialize the Raft server
	if err := s.setupRaft(); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start Raft", "error", err)
		return nil, fmt.Errorf("Failed to start Raft: %v", err)
	}

	// Initialize the wan Serf
	s.serf, err = s.setupSerf(config.SerfConfig, s.eventCh, serfSnapshot)
	if err != nil {
		s.Shutdown()
		s.logger.Error("failed to start serf WAN", "error", err)
		return nil, fmt.Errorf("Failed to start serf: %v", err)
	}

	// Initialize the scheduling workers
	if err := s.setupWorkers(s.shutdownCtx); err != nil {
		s.Shutdown()
		s.logger.Error("failed to start workers", "error", err)
		return nil, fmt.Errorf("Failed to start workers: %v", err)
	}

	// Setup the Consul syncer
	if err := s.setupConsulSyncer(); err != nil {
		s.logger.Error("failed to create server consul syncer", "error", err)
		return nil, fmt.Errorf("failed to create server Consul syncer: %v", err)
	}

	// Setup the deployment watcher.
	if err := s.setupDeploymentWatcher(); err != nil {
		s.logger.Error("failed to create deployment watcher", "error", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped cause: if bind failed, free the serf port (default 4648) or change ports.serf
2. Fix advertise_addr/bind_addr to a valid local IP
3. Delete or repair the corrupt serf snapshot under data_dir/serf if state is damaged
4. Confirm host networking allows UDP/TCP gossip traffic

Example fix

// before
ports { serf = 4648 } # conflict with second agent on same host
// after
ports { serf = 14648 }
advertise { serf = "10.0.0.5" }

// Pre-flight: serf port free and advertise address local
if ln, err := net.Listen("tcp", fmt.Sprintf("%s:%d", cfg.BindAddr, cfg.Ports.Serf)); err != nil {
    return fmt.Errorf("serf port in use: %w", err)
} else { ln.Close() }
if ip := net.ParseIP(cfg.AdvertiseAddrs.Serf); ip == nil {
    return fmt.Errorf("invalid serf advertise address")
}

srv, err := nomad.NewServer(config, catalog, consulFn)
if err != nil {
    if strings.Contains(err.Error(), "Failed to start serf") {
        // inspect cause: bind conflict, bad advertise addr, corrupt snapshot
    }
    return err
}

When it happens

Trigger: Starting a server where setupSerf(config.SerfConfig, s.eventCh, serfSnapshot) fails — gossip port in use, invalid advertise/bind address, unreadable or corrupt serf snapshot in data_dir, or LAN Serf setup issues surfaced via the same wrapper.

Common situations: Port 4648 (serf) already used by another agent or process; advertise_addr pointing at an interface that doesn't exist; leftover serf/snapshot state in data_dir from a previous failed run; multicast/network restrictions in some container environments.

Related errors

• server setup failed: %v
• deprecated config field 'RaftBoltNoFreelistSync' is set; use
• Failed to start RPC layer: %v
• Failed to start Raft: %v
• Failed to start workers: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6578dc1c9a64e93a. Report an issue: GitHub.