failed to transfer leadership in %d attempts. last error: %w

Error message

failed to transfer leadership in %d attempts. last error: %w

What it means

After exhausting its retry budget (default 3 attempts), leadershipTransfer wraps the last Raft error in this message. It indicates repeated failed attempts to hand off leadership, with the underlying cause appended via %w.

Source

Thrown at nomad/leader.go:219

		if err == raft.ErrRaftShutdown {
			return err
		}

		// ErrUnsupportedProtocol: Don't retry if the Raft version doesn't
		// support leadership transfer since this will never succeed.
		if err == raft.ErrUnsupportedProtocol {
			return fmt.Errorf("leadership transfer not supported with Raft version lower than 3")
		}

		// ErrEnqueueTimeout: This seems to be the valid time to retry.
		s.logger.Error("failed to transfer leadership attempt, will retry",
			"attempt", i,
			"retry_limit", retryCount,
			"error", err,
		)
		lastError = err
	}
	return fmt.Errorf("failed to transfer leadership in %d attempts. last error: %w", retryCount, lastError)
}

func (s *Server) leadershipTransfer() error {
	retryCount := 3
	for i := range retryCount {
		err := s.raft.LeadershipTransfer().Error()
		if err == nil {
			s.logger.Info("successfully transferred leadership")
			return nil
		}

		// Don't retry if the Raft version doesn't support leadership transfer
		// since this will never succeed.
		if err == raft.ErrUnsupportedProtocol {
			return fmt.Errorf("leadership transfer not supported with Raft version lower than 3")
		}

		s.logger.Error("failed to transfer leadership attempt, will retry",

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the wrapped last error to identify the root cause and fix that (connectivity, log lag, Raft version).
Ensure at least one healthy, up-to-date voter server exists to receive leadership.
Restart the current leader to force an election rather than a graceful transfer if transfers keep failing.
Verify cluster peer health (`nomad server members`, Raft stats) before retrying.

Example fix

// before: repeatedly retrying transfers against a single-node cluster
// after: add a second/third server, wait for it to become a healthy voter, then retry transfer
nomad server members   # confirm >1 healthy voter
nomad operator raft list-peers

Defensive patterns

Strategy: retry

Validate before calling

peers := raftHealthyVoters() // ensure >1 healthy, up-to-date voter before attempting transfer
if len(peers) < 2 {
    return fmt.Errorf("need at least one healthy peer to receive leadership")
}

Try / catch

err := transferLeadership()
if err != nil && strings.Contains(err.Error(), "failed to transfer leadership in") {
    var lastErr error
    errors.As(err, &lastErr) // inspect wrapped cause
    log.Printf("transfer exhausted retries: %v; consider restarting leader", lastErr)
}

Prevention

Monitor Raft commit indexes and peer health before transfers.
Keep an odd number of healthy voter servers.
Use errors.As to inspect the wrapped last error for the real cause.
If transfers repeatedly fail, restart the leader to force an election instead.

When it happens

Trigger: s.raft.LeadershipTransfer() fails on every one of the 3 attempts — e.g. cluster lacks a viable transfer target, target server is behind on logs, or ErrUnsupportedProtocol conditions persist.

Common situations: Rolling restarts where only one server remains (no transferable peer); a lagging voter that can never win an election; degraded networks causing transfer timeouts on each attempt.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/45497e2d54d3d264. Report an issue: GitHub.