{"record":{"id":"0abef8ff42e13d31","repo":"hashicorp/nomad","slug":"timed-out-waiting-for-re-run-of-leader-actions","errorCode":null,"errorMessage":"timed out waiting for re-run of leader actions","messagePattern":"timed out waiting for re-run of leader actions","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"nomad/operator_endpoint.go","lineNumber":737,"sourceCode":"\tcase <-timeoutCh:\n\t\thandleFailure(500, fmt.Errorf(\"timed out waiting to re-run leader actions\"))\n\n\t// Make sure we don't get stuck during shutdown\n\tcase <-op.srv.shutdownCh:\n\t}\n\n\tselect {\n\t// Wait for the leader loop to finish up.\n\tcase err := <-lerrCh:\n\t\tif err != nil {\n\t\t\thandleFailure(500, err)\n\t\t\treturn\n\t\t}\n\n\t// We might have lost leadership while the loop was doing its\n\t// thing.\n\tcase <-timeoutCh:\n\t\thandleFailure(500, fmt.Errorf(\"timed out waiting for re-run of leader actions\"))\n\n\t// Make sure we don't get stuck during shutdown\n\tcase <-op.srv.shutdownCh:\n\t}\n\n\treply.Index, _ = op.srv.State().LatestIndex()\n\top.srv.setQueryMeta(&reply.QueryMeta)\n\tencoder.Encode(reply)\n}\n\nfunc (op *Operator) UpgradeCheckVaultWorkloadIdentity(\n\targs *structs.UpgradeCheckVaultWorkloadIdentityRequest,\n\treply *structs.UpgradeCheckVaultWorkloadIdentityResponse,\n) error {\n\tauthErr := op.srv.Authenticate(op.ctx, args)\n\tif done, err := op.srv.forward(\"Operator.UpgradeCheckVaultWorkloadIdentity\", args, args, reply); done {\n\t\treturn err\n\t}","sourceCodeStart":719,"sourceCodeEnd":755,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/operator_endpoint.go#L719-L755","documentation":"After the leader loop accepts the reassert request, snapshotRestore waits (second select) for the loop to finish re-running leader actions, again bounded by the 1-minute timeoutCh. If the loop takes too long or leadership was lost mid-loop, this 500 error is returned.","triggerScenarios":"reassertLeader processing (rebuilding leader state from the new state store) exceeds 1 minute, or leadership is lost while the loop is running, so the timeoutCh case fires instead of lerrCh closing.","commonSituations":"Very large restored snapshot making post-restore leader work (reaping, scheduling, vault/token revocation setup) slow; leadership flapping on a degraded cluster.","solutions":["Retry when the cluster has a stable leader and lower load","Scale/investigate the leader loop slowness — large state stores may need more time or faster hardware","Check server logs for the reassert leader loop duration and any panics","Re-attempt the restore during a maintenance window"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"leader, err := client.Status().Leader(nil)\nif err != nil || leader == \"\" { return errors.New(\"no stable leader\") }","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"timed out waiting for re-run of leader actions\") {\n    // leader loop too slow or lost leadership: retry in maintenance window\n}","preventionTips":["Restore large snapshots during maintenance windows","Ensure the leader has sufficient CPU for post-restore leader work","Monitor leader loop durations in server metrics"],"tags":["leadership","timeout","raft","snapshot"],"backgroundTag":"lost-leadership-during-operation","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}