{"record":{"id":"dd104c6aefb1f42a","repo":"hashicorp/nomad","slug":"index-update-failed-v-dd104c","errorCode":null,"errorMessage":"index update failed: %v","messagePattern":"index update failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1386,"sourceCode":"\texisting, err := txn.First(\"nodes\", \"id\", nodeID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"node lookup failed: %v\", err)\n\t}\n\tif existing == nil {\n\t\treturn fmt.Errorf(\"node not found\")\n\t}\n\n\t// Copy the existing node\n\texistingNode := existing.(*structs.Node)\n\tcopyNode := existingNode.Copy()\n\tappendNodeEvents(index, copyNode, events)\n\n\t// Insert the node\n\tif err := txn.Insert(\"nodes\", copyNode); err != nil {\n\t\treturn fmt.Errorf(\"node update failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"nodes\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// appendNodeEvents is a helper that takes a node and new events and appends\n// them, pruning older events as needed.\nfunc appendNodeEvents(index uint64, node *structs.Node, events []*structs.NodeEvent) {\n\t// Add the events, updating the indexes\n\tfor _, e := range events {\n\t\te.CreateIndex = index\n\t\tnode.Events = append(node.Events, e)\n\t}\n\n\t// Keep node events pruned to not exceed the max allowed\n\tif l := len(node.Events); l > structs.MaxRetainedNodeEvents {\n\t\tdelta := l - structs.MaxRetainedNodeEvents\n\t\tnode.Events = node.Events[delta:]","sourceCodeStart":1368,"sourceCodeEnd":1404,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1368-L1404","documentation":"This error wraps a memdb table-insert failure when updating the node's modify index in the \"index\" table during a node-event upsert transaction in Nomad's state store. It indicates the Raft-replicated state mutation could not record its index entry, so the whole transaction is aborted and the Raft log entry fails. It is an internal consistency failure, not something caused by user input.","triggerScenarios":"Calling UpsertNodeEvents (via the Node.UpdateStatus / event batching path) when the underlying memdb transaction cannot insert the IndexEntry{TableNodeEvents... \"nodes\"} row — typically memdb corruption, an invalid table/index name, or transaction already in a failed state after the preceding nodes insert.","commonSituations":"Seen in operator logs as a failed Raft write when node heartbeat/status updates flush events; usually accompanies memory pressure, disk issues on the Raft layer, or a bug report rather than routine misconfiguration.","solutions":["Retry the operation — Raft apply failures are often transient; the client re-registers node events on the next heartbeat.","Check server logs for an earlier 'node update failed' error in the same transaction; fix the root cause there.","Verify the Nomad version for known memdb/raft bugs and upgrade to the latest patch release.","If persistent, restart the server agent and, if Raft state is corrupt, restore from a backup as documented in Nomad's recovery guide."],"exampleFix":"// before: error surfaces from an internal txn\n// nomad/state/state_store.go (maintainer fix path)\nif err := txn.Insert(\"index\", &IndexEntry{\"nodes\", index}); err != nil {\n\treturn fmt.Errorf(\"index update failed: %v\", err)\n}\n// after: caller side — treat as transient and retry the RPC\nif err := client.Nodes().UpdateStatus(...); err != nil && strings.Contains(err.Error(), \"index update failed\") {\n\ttime.Sleep(backoff)\n\treturn client.Nodes().UpdateStatus(...)\n}","handlingStrategy":"retry","validationCode":"// preflight: ensure the node is registered and server is leader-reachable\n_, _, err := client.Nodes().Info(nodeID, nil)\nif err != nil { return fmt.Errorf(\"node unreachable, skip event update: %w\", err) }","typeGuard":null,"tryCatchPattern":"err := client.Nodes().UpdateStatus(nodeID, \"ready\", nil)\nif err != nil && strings.Contains(err.Error(), \"index update failed\") {\n    // transient raft/state-store write failure: backoff and retry\n    time.Sleep(2 * time.Second)\n    return retryUpdate(nodeID)\n}\nreturn err","preventionTips":["Nomad retries node event flushes on the next heartbeat; avoid tight manual retry loops.","Monitor server memory and Raft store health.","Keep Nomad servers on the latest patch release.","Alert on any 'index update failed' occurrences in server logs."],"tags":["nomad","state-store","raft","memdb"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}