{"record":{"id":"6578dc1c9a64e93a","repo":"hashicorp/nomad","slug":"failed-to-start-serf-v","errorCode":null,"errorMessage":"Failed to start serf: %v","messagePattern":"Failed to start serf: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/server.go","lineNumber":499,"sourceCode":"\t\tAclsEnabled:    s.config.ACLEnabled,\n\t\tVerifyTLS:      s.config.TLSConfig != nil && s.config.TLSConfig.EnableRPC && s.config.TLSConfig.VerifyServerHostname,\n\t\tRegion:         s.Region(),\n\t\tEncrypter:      s.encrypter,\n\t})\n\n\t// Initialize the Raft server\n\tif err := s.setupRaft(); err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start Raft\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start Raft: %v\", err)\n\t}\n\n\t// Initialize the wan Serf\n\ts.serf, err = s.setupSerf(config.SerfConfig, s.eventCh, serfSnapshot)\n\tif err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start serf WAN\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start serf: %v\", err)\n\t}\n\n\t// Initialize the scheduling workers\n\tif err := s.setupWorkers(s.shutdownCtx); err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start workers\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start workers: %v\", err)\n\t}\n\n\t// Setup the Consul syncer\n\tif err := s.setupConsulSyncer(); err != nil {\n\t\ts.logger.Error(\"failed to create server consul syncer\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"failed to create server Consul syncer: %v\", err)\n\t}\n\n\t// Setup the deployment watcher.\n\tif err := s.setupDeploymentWatcher(); err != nil {\n\t\ts.logger.Error(\"failed to create deployment watcher\", \"error\", err)","sourceCodeStart":481,"sourceCodeEnd":517,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L481-L517","documentation":"NewServer wraps failures from s.setupSerf for the WAN Serf membership layer with \"Failed to start serf: %v\" after calling Shutdown. Serf setup creates the membership and event handlers and binds its gossip port, so failures are usually bind conflicts, bad advertise addresses, or a corrupt Serf snapshot.","triggerScenarios":"Starting a server where setupSerf(config.SerfConfig, s.eventCh, serfSnapshot) fails — gossip port in use, invalid advertise/bind address, unreadable or corrupt serf snapshot in data_dir, or LAN Serf setup issues surfaced via the same wrapper.","commonSituations":"Port 4648 (serf) already used by another agent or process; advertise_addr pointing at an interface that doesn't exist; leftover serf/snapshot state in data_dir from a previous failed run; multicast/network restrictions in some container environments.","solutions":["Check the wrapped cause: if bind failed, free the serf port (default 4648) or change ports.serf","Fix advertise_addr/bind_addr to a valid local IP","Delete or repair the corrupt serf snapshot under data_dir/serf if state is damaged","Confirm host networking allows UDP/TCP gossip traffic"],"exampleFix":"// before\nports { serf = 4648 } # conflict with second agent on same host\n// after\nports { serf = 14648 }\nadvertise { serf = \"10.0.0.5\" }","handlingStrategy":"validation","validationCode":"// Pre-flight: serf port free and advertise address local\nif ln, err := net.Listen(\"tcp\", fmt.Sprintf(\"%s:%d\", cfg.BindAddr, cfg.Ports.Serf)); err != nil {\n    return fmt.Errorf(\"serf port in use: %w\", err)\n} else { ln.Close() }\nif ip := net.ParseIP(cfg.AdvertiseAddrs.Serf); ip == nil {\n    return fmt.Errorf(\"invalid serf advertise address\")\n}","typeGuard":null,"tryCatchPattern":"srv, err := nomad.NewServer(config, catalog, consulFn)\nif err != nil {\n    if strings.Contains(err.Error(), \"Failed to start serf\") {\n        // inspect cause: bind conflict, bad advertise addr, corrupt snapshot\n    }\n    return err\n}","preventionTips":["Reserve distinct serf ports when running multiple agents per host","Validate advertise addresses resolve to existing local interfaces","Clean leftover data_dir/serf snapshots from failed runs before restart"],"tags":["nomad","go","serf","gossip","startup"],"backgroundTag":"port-already-in-use","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}