hashicorp/nomad · error

timed out waiting for re-run of leader actions

Error message

timed out waiting for re-run of leader actions

What it means

After the leader loop accepts the reassert request, snapshotRestore waits (second select) for the loop to finish re-running leader actions, again bounded by the 1-minute timeoutCh. If the loop takes too long or leadership was lost mid-loop, this 500 error is returned.

Source

Thrown at nomad/operator_endpoint.go:737

	case <-timeoutCh:
		handleFailure(500, fmt.Errorf("timed out waiting to re-run leader actions"))

	// Make sure we don't get stuck during shutdown
	case <-op.srv.shutdownCh:
	}

	select {
	// Wait for the leader loop to finish up.
	case err := <-lerrCh:
		if err != nil {
			handleFailure(500, err)
			return
		}

	// We might have lost leadership while the loop was doing its
	// thing.
	case <-timeoutCh:
		handleFailure(500, fmt.Errorf("timed out waiting for re-run of leader actions"))

	// Make sure we don't get stuck during shutdown
	case <-op.srv.shutdownCh:
	}

	reply.Index, _ = op.srv.State().LatestIndex()
	op.srv.setQueryMeta(&reply.QueryMeta)
	encoder.Encode(reply)
}

func (op *Operator) UpgradeCheckVaultWorkloadIdentity(
	args *structs.UpgradeCheckVaultWorkloadIdentityRequest,
	reply *structs.UpgradeCheckVaultWorkloadIdentityResponse,
) error {
	authErr := op.srv.Authenticate(op.ctx, args)
	if done, err := op.srv.forward("Operator.UpgradeCheckVaultWorkloadIdentity", args, args, reply); done {
		return err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry when the cluster has a stable leader and lower load
2. Scale/investigate the leader loop slowness — large state stores may need more time or faster hardware
3. Check server logs for the reassert leader loop duration and any panics
4. Re-attempt the restore during a maintenance window

leader, err := client.Status().Leader(nil)
if err != nil || leader == "" { return errors.New("no stable leader") }

if err != nil && strings.Contains(err.Error(), "timed out waiting for re-run of leader actions") {
    // leader loop too slow or lost leadership: retry in maintenance window
}

When it happens

Trigger: reassertLeader processing (rebuilding leader state from the new state store) exceeds 1 minute, or leadership is lost while the loop is running, so the timeoutCh case fires instead of lerrCh closing.

Common situations: Very large restored snapshot making post-restore leader work (reaping, scheduling, vault/token revocation setup) slow; leadership flapping on a degraded cluster.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• timed out waiting to re-run leader actions
• CSI.ControllerListSnapshots: %v
• failed to read snapshot: %w
• failed to open snapshot dir: %v
• Failed to load snapshot from archive: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0abef8ff42e13d31. Report an issue: GitHub.