hashicorp/nomad · error

failed to transfer leadership in

Error message

failed to transfer leadership in %d attempts. last error: %w

What it means

After exhausting its retry budget (default 3 attempts), leadershipTransfer wraps the last Raft error in this message. It indicates repeated failed attempts to hand off leadership, with the underlying cause appended via %w.

Solutions

  1. Read the wrapped last error to identify the root cause and fix that (connectivity, log lag, Raft version).
  2. Ensure at least one healthy, up-to-date voter server exists to receive leadership.
  3. Restart the current leader to force an election rather than a graceful transfer if transfers keep failing.
  4. Verify cluster peer health (`nomad server members`, Raft stats) before retrying.

Example fix

// before: repeatedly retrying transfers against a single-node cluster
// after: add a second/third server, wait for it to become a healthy voter, then retry transfer
nomad server members   # confirm >1 healthy voter
nomad operator raft list-peers
Defensive patterns

Strategy: retry

Validate before calling

peers := raftHealthyVoters() // ensure >1 healthy, up-to-date voter before attempting transfer
if len(peers) < 2 {
    return fmt.Errorf("need at least one healthy peer to receive leadership")
}

Try / catch

err := transferLeadership()
if err != nil && strings.Contains(err.Error(), "failed to transfer leadership in") {
    var lastErr error
    errors.As(err, &lastErr) // inspect wrapped cause
    log.Printf("transfer exhausted retries: %v; consider restarting leader", lastErr)
}

Prevention

When it happens

Trigger: s.raft.LeadershipTransfer() fails on every one of the 3 attempts — e.g. cluster lacks a viable transfer target, target server is behind on logs, or ErrUnsupportedProtocol conditions persist.

Common situations: Rolling restarts where only one server remains (no transferable peer); a lagging voter that can never win an election; degraded networks causing transfer timeouts on each attempt.

Related errors


AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/45497e2d54d3d264. Report an issue: GitHub.

Appendix: source

Thrown at nomad/leader.go:219

		if err == raft.ErrRaftShutdown {
			return err
		}

		// ErrUnsupportedProtocol: Don't retry if the Raft version doesn't
		// support leadership transfer since this will never succeed.
		if err == raft.ErrUnsupportedProtocol {
			return fmt.Errorf("leadership transfer not supported with Raft version lower than 3")
		}

		// ErrEnqueueTimeout: This seems to be the valid time to retry.
		s.logger.Error("failed to transfer leadership attempt, will retry",
			"attempt", i,
			"retry_limit", retryCount,
			"error", err,
		)
		lastError = err
	}
	return fmt.Errorf("failed to transfer leadership in %d attempts. last error: %w", retryCount, lastError)
}

func (s *Server) leadershipTransfer() error {
	retryCount := 3
	for i := range retryCount {
		err := s.raft.LeadershipTransfer().Error()
		if err == nil {
			s.logger.Info("successfully transferred leadership")
			return nil
		}

		// Don't retry if the Raft version doesn't support leadership transfer
		// since this will never succeed.
		if err == raft.ErrUnsupportedProtocol {
			return fmt.Errorf("leadership transfer not supported with Raft version lower than 3")
		}

		s.logger.Error("failed to transfer leadership attempt, will retry",

View on GitHub (pinned to 482b49bf1a)