{"record":{"id":"45497e2d54d3d264","repo":"hashicorp/nomad","slug":"failed-to-transfer-leadership-in-d-attempts-last","errorCode":null,"errorMessage":"failed to transfer leadership in %d attempts. last error: %w","messagePattern":"failed to transfer leadership in (.+?) attempts\\. last error: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/leader.go","lineNumber":219,"sourceCode":"\t\tif err == raft.ErrRaftShutdown {\n\t\t\treturn err\n\t\t}\n\n\t\t// ErrUnsupportedProtocol: Don't retry if the Raft version doesn't\n\t\t// support leadership transfer since this will never succeed.\n\t\tif err == raft.ErrUnsupportedProtocol {\n\t\t\treturn fmt.Errorf(\"leadership transfer not supported with Raft version lower than 3\")\n\t\t}\n\n\t\t// ErrEnqueueTimeout: This seems to be the valid time to retry.\n\t\ts.logger.Error(\"failed to transfer leadership attempt, will retry\",\n\t\t\t\"attempt\", i,\n\t\t\t\"retry_limit\", retryCount,\n\t\t\t\"error\", err,\n\t\t)\n\t\tlastError = err\n\t}\n\treturn fmt.Errorf(\"failed to transfer leadership in %d attempts. last error: %w\", retryCount, lastError)\n}\n\nfunc (s *Server) leadershipTransfer() error {\n\tretryCount := 3\n\tfor i := range retryCount {\n\t\terr := s.raft.LeadershipTransfer().Error()\n\t\tif err == nil {\n\t\t\ts.logger.Info(\"successfully transferred leadership\")\n\t\t\treturn nil\n\t\t}\n\n\t\t// Don't retry if the Raft version doesn't support leadership transfer\n\t\t// since this will never succeed.\n\t\tif err == raft.ErrUnsupportedProtocol {\n\t\t\treturn fmt.Errorf(\"leadership transfer not supported with Raft version lower than 3\")\n\t\t}\n\n\t\ts.logger.Error(\"failed to transfer leadership attempt, will retry\",","sourceCodeStart":201,"sourceCodeEnd":237,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/leader.go#L201-L237","documentation":"After exhausting its retry budget (default 3 attempts), leadershipTransfer wraps the last Raft error in this message. It indicates repeated failed attempts to hand off leadership, with the underlying cause appended via %w.","triggerScenarios":"s.raft.LeadershipTransfer() fails on every one of the 3 attempts — e.g. cluster lacks a viable transfer target, target server is behind on logs, or ErrUnsupportedProtocol conditions persist.","commonSituations":"Rolling restarts where only one server remains (no transferable peer); a lagging voter that can never win an election; degraded networks causing transfer timeouts on each attempt.","solutions":["Read the wrapped last error to identify the root cause and fix that (connectivity, log lag, Raft version).","Ensure at least one healthy, up-to-date voter server exists to receive leadership.","Restart the current leader to force an election rather than a graceful transfer if transfers keep failing.","Verify cluster peer health (`nomad server members`, Raft stats) before retrying."],"exampleFix":"// before: repeatedly retrying transfers against a single-node cluster\n// after: add a second/third server, wait for it to become a healthy voter, then retry transfer\nnomad server members   # confirm >1 healthy voter\nnomad operator raft list-peers","handlingStrategy":"retry","validationCode":"peers := raftHealthyVoters() // ensure >1 healthy, up-to-date voter before attempting transfer\nif len(peers) < 2 {\n    return fmt.Errorf(\"need at least one healthy peer to receive leadership\")\n}","typeGuard":null,"tryCatchPattern":"err := transferLeadership()\nif err != nil && strings.Contains(err.Error(), \"failed to transfer leadership in\") {\n    var lastErr error\n    errors.As(err, &lastErr) // inspect wrapped cause\n    log.Printf(\"transfer exhausted retries: %v; consider restarting leader\", lastErr)\n}","preventionTips":["Monitor Raft commit indexes and peer health before transfers.","Keep an odd number of healthy voter servers.","Use errors.As to inspect the wrapped last error for the real cause.","If transfers repeatedly fail, restart the leader to force an election instead."],"tags":["nomad","raft","leadership","retry-exhausted"],"backgroundTag":"leadership-transfer-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}