{"record":{"id":"a6d74c3d9fa1a9c7","repo":"hashicorp/nomad","slug":"timed-out-waiting-to-re-run-leader-actions","errorCode":null,"errorMessage":"timed out waiting to re-run leader actions","messagePattern":"timed out waiting to re-run leader actions","errorType":"http","errorClass":null,"httpStatus":500,"severity":"error","filePath":"nomad/operator_endpoint.go","lineNumber":720,"sourceCode":"\terr = <-errCh\n\tif err != nil {\n\t\thandleFailure(400, fmt.Errorf(\"failed to read stream: %v\", err))\n\t\treturn\n\t}\n\n\t// This'll be used for feedback from the leader loop.\n\ttimeoutCh := time.After(time.Minute)\n\n\tlerrCh := make(chan error, 1)\n\n\tselect {\n\t// Reassert leader actions and update all leader related state\n\t// with new state store content.\n\tcase op.srv.reassertLeaderCh <- lerrCh:\n\n\t// We might have lost leadership while waiting to kick the loop.\n\tcase <-timeoutCh:\n\t\thandleFailure(500, fmt.Errorf(\"timed out waiting to re-run leader actions\"))\n\n\t// Make sure we don't get stuck during shutdown\n\tcase <-op.srv.shutdownCh:\n\t}\n\n\tselect {\n\t// Wait for the leader loop to finish up.\n\tcase err := <-lerrCh:\n\t\tif err != nil {\n\t\t\thandleFailure(500, err)\n\t\t\treturn\n\t\t}\n\n\t// We might have lost leadership while the loop was doing its\n\t// thing.\n\tcase <-timeoutCh:\n\t\thandleFailure(500, fmt.Errorf(\"timed out waiting for re-run of leader actions\"))\n","sourceCodeStart":702,"sourceCodeEnd":738,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/operator_endpoint.go#L702-L738","documentation":"snapshotRestore waits up to 1 minute to send a reassert request to the leader loop (op.srv.reassertLeaderCh) after restoring state. If the leader loop does not consume it within the timeout, the server assumes leadership was lost and returns this 500 error.","triggerScenarios":"Leadership was lost (or leadership transition is stalled) between restoring the snapshot and pushing the reassertLeaderCh notification; the select's timeoutCh case fires.","commonSituations":"Cluster election happened during the restore window; leader step-down triggered by Raft changes; slow leader loop unable to process the channel within 60s.","solutions":["Confirm the node is still the leader (nomad operator raft list-peers) and re-run the restore on the current leader","Investigate why leadership flapped during the restore (Raft logs, elections)","Retry the restore once leadership is stable","Increase patience/window only if the leader loop is known to be slow — otherwise treat as a real leadership loss"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"leader, err := client.Status().Leader(nil)\nif err != nil || leader == \"\" { return errors.New(\"cluster has no leader; restore would fail\") }","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"timed out waiting to re-run leader actions\") {\n    // leadership lost: wait for stable leader, then retry restore\n}","preventionTips":["Perform restores during quiet periods with a stable leader","Check raft elections/step-downs in logs before retrying","Retry only after leadership is confirmed stable"],"tags":["leadership","raft","timeout","snapshot"],"backgroundTag":"lost-leadership-during-operation","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}