{"record":{"id":"04204e6f2a1b10fa","repo":"hashicorp/nomad","slug":"failed-to-start-raft-v","errorCode":null,"errorMessage":"Failed to start Raft: %v","messagePattern":"Failed to start Raft: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"nomad/server.go","lineNumber":491,"sourceCode":"\t\ts.logger.Error(\"failed to start RPC layer\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start RPC layer: %v\", err)\n\t}\n\n\ts.auth = auth.NewAuthenticator(&auth.AuthenticatorConfig{\n\t\tStateFn:        s.State,\n\t\tLogger:         s.logger,\n\t\tGetLeaderACLFn: s.getLeaderAcl,\n\t\tAclsEnabled:    s.config.ACLEnabled,\n\t\tVerifyTLS:      s.config.TLSConfig != nil && s.config.TLSConfig.EnableRPC && s.config.TLSConfig.VerifyServerHostname,\n\t\tRegion:         s.Region(),\n\t\tEncrypter:      s.encrypter,\n\t})\n\n\t// Initialize the Raft server\n\tif err := s.setupRaft(); err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start Raft\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start Raft: %v\", err)\n\t}\n\n\t// Initialize the wan Serf\n\ts.serf, err = s.setupSerf(config.SerfConfig, s.eventCh, serfSnapshot)\n\tif err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start serf WAN\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start serf: %v\", err)\n\t}\n\n\t// Initialize the scheduling workers\n\tif err := s.setupWorkers(s.shutdownCtx); err != nil {\n\t\ts.Shutdown()\n\t\ts.logger.Error(\"failed to start workers\", \"error\", err)\n\t\treturn nil, fmt.Errorf(\"Failed to start workers: %v\", err)\n\t}\n\n\t// Setup the Consul syncer","sourceCodeStart":473,"sourceCodeEnd":509,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/server.go#L473-L509","documentation":"If s.setupRaft() fails while NewServer is initializing consensus, the server shuts down and returns \"Failed to start Raft: %v\". setupRaft covers transport setup, log store creation, snapshot store creation, and raft bootstrap, so the embedded cause identifies the failing layer.","triggerScenarios":"Starting a server where Raft initialization fails: unwritable data_dir (raft.db/snapshots), corrupted raft state, BoltDB log store open failure, transport bind failure on the raft port, or invalid bootstrap configuration.","commonSituations":"Disk full or permissions on data_dir preventing raft.db creation; corrupted BoltDB after a crash; raft port (4647) conflict; mismatched server IDs/state after cloning a VM image with existing data_dir.","solutions":["Read the wrapped cause and fix accordingly (permissions, disk space, port conflict)","If raft state is corrupted, restore data_dir from backup or (accepting data loss) remove raft.db on a non-quorum node","Ensure each server has a unique node_id and never clone data_dir between servers","Verify data_dir exists and is writable by the nomad user"],"exampleFix":"// before\nchown -R root:root /var/lib/nomad\n// after\nchown -R nomad:nomad /var/lib/nomad && systemctl restart nomad","handlingStrategy":"validation","validationCode":"// Pre-flight Raft prerequisites: writable data_dir and free raft port\nif err := os.MkdirAll(cfg.DataDir, 0o700); err != nil {\n    return fmt.Errorf(\"data_dir not writable: %w\", err)\n}\nif f, err := os.Create(filepath.Join(cfg.DataDir, \".write-test\")); err != nil {\n    return err\n} else { f.Close(); os.Remove(f.Name()) }","typeGuard":null,"tryCatchPattern":"srv, err := nomad.NewServer(config, catalog, consulFn)\nif err != nil {\n    if strings.Contains(err.Error(), \"Failed to start Raft\") {\n        // check data_dir permissions/disk and raft.db integrity before retry\n    }\n    return err\n}","preventionTips":["Ensure data_dir is owned by the nomad user with adequate disk space","Never clone data_dir between servers; keep unique node_id per server","Monitor raft.db integrity after unclean shutdowns; keep backups"],"tags":["nomad","go","raft","consensus","startup"],"backgroundTag":"raft-startup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}