hashicorp/nomad · error

timed out waiting to re-run leader actions

Error message

timed out waiting to re-run leader actions

What it means

snapshotRestore waits up to 1 minute to send a reassert request to the leader loop (op.srv.reassertLeaderCh) after restoring state. If the leader loop does not consume it within the timeout, the server assumes leadership was lost and returns this 500 error.

Source

Thrown at nomad/operator_endpoint.go:720

	err = <-errCh
	if err != nil {
		handleFailure(400, fmt.Errorf("failed to read stream: %v", err))
		return
	}

	// This'll be used for feedback from the leader loop.
	timeoutCh := time.After(time.Minute)

	lerrCh := make(chan error, 1)

	select {
	// Reassert leader actions and update all leader related state
	// with new state store content.
	case op.srv.reassertLeaderCh <- lerrCh:

	// We might have lost leadership while waiting to kick the loop.
	case <-timeoutCh:
		handleFailure(500, fmt.Errorf("timed out waiting to re-run leader actions"))

	// Make sure we don't get stuck during shutdown
	case <-op.srv.shutdownCh:
	}

	select {
	// Wait for the leader loop to finish up.
	case err := <-lerrCh:
		if err != nil {
			handleFailure(500, err)
			return
		}

	// We might have lost leadership while the loop was doing its
	// thing.
	case <-timeoutCh:
		handleFailure(500, fmt.Errorf("timed out waiting for re-run of leader actions"))

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm the node is still the leader (nomad operator raft list-peers) and re-run the restore on the current leader
2. Investigate why leadership flapped during the restore (Raft logs, elections)
3. Retry the restore once leadership is stable
4. Increase patience/window only if the leader loop is known to be slow — otherwise treat as a real leadership loss

leader, err := client.Status().Leader(nil)
if err != nil || leader == "" { return errors.New("cluster has no leader; restore would fail") }

if err != nil && strings.Contains(err.Error(), "timed out waiting to re-run leader actions") {
    // leadership lost: wait for stable leader, then retry restore
}

When it happens

Trigger: Leadership was lost (or leadership transition is stalled) between restoring the snapshot and pushing the reassertLeaderCh notification; the select's timeoutCh case fires.

Common situations: Cluster election happened during the restore window; leader step-down triggered by Raft changes; slow leader loop unable to process the channel within 60s.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• timed out waiting for re-run of leader actions
• CSI.ControllerListSnapshots: %v
• failed to read snapshot: %w
• failed to open snapshot dir: %v
• Failed to load snapshot from archive: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a6d74c3d9fa1a9c7. Report an issue: GitHub.