{"record":{"id":"a2804b9ceaf9b881","repo":"hashicorp/nomad","slug":"node-update-failed-v","errorCode":null,"errorMessage":"node update failed: %v","messagePattern":"node update failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1383,"sourceCode":"// older events once this bound has been reached.\nfunc (s *StateStore) upsertNodeEvents(index uint64, nodeID string, events []*structs.NodeEvent, txn *txn) error {\n\t// Lookup the node\n\texisting, err := txn.First(\"nodes\", \"id\", nodeID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"node lookup failed: %v\", err)\n\t}\n\tif existing == nil {\n\t\treturn fmt.Errorf(\"node not found\")\n\t}\n\n\t// Copy the existing node\n\texistingNode := existing.(*structs.Node)\n\tcopyNode := existingNode.Copy()\n\tappendNodeEvents(index, copyNode, events)\n\n\t// Insert the node\n\tif err := txn.Insert(\"nodes\", copyNode); err != nil {\n\t\treturn fmt.Errorf(\"node update failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"nodes\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// appendNodeEvents is a helper that takes a node and new events and appends\n// them, pruning older events as needed.\nfunc appendNodeEvents(index uint64, node *structs.Node, events []*structs.NodeEvent) {\n\t// Add the events, updating the indexes\n\tfor _, e := range events {\n\t\te.CreateIndex = index\n\t\tnode.Events = append(node.Events, e)\n\t}\n\n\t// Keep node events pruned to not exceed the max allowed","sourceCodeStart":1365,"sourceCodeEnd":1401,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1365-L1401","documentation":"upsertNodeEvents wraps a failure from txn.Insert(\"nodes\", copyNode) when re-inserting the mutated node record. This indicates the in-transaction write failed (memdb/bolt insert error), not that the node is missing. Raised through StateStore.UpsertNodeEvents.","triggerScenarios":"txn.Insert fails during a node event update — caused by underlying store errors, transaction state issues, or attempting to insert into an aborted/invalid transaction within a composite write (e.g. inside UpdateNodeStatus/Drain transactions).","commonSituations":"Disk/full or corrupted bolt backend during event-heavy workloads; a nested transaction abort earlier in the call chain making subsequent inserts fail; internal bugs in the write path.","solutions":["Inspect the wrapped error (%v) for the underlying memdb/bolt cause.","Retry the operation after the transaction is healthy; ensure the txn is not aborted before this insert.","Restart the server agent / restore from snapshot if the state store is corrupted.","Check disk space and file permissions on the Nomad data directory."],"exampleFix":"// before\nif err := txn.Insert(\"nodes\", copyNode); err != nil {\n    return fmt.Errorf(\"node update failed: %v\", err)\n}\n// after\nif err := txn.Insert(\"nodes\", copyNode); err != nil {\n    return fmt.Errorf(\"node update failed (node %s): %w\", nodeID, err)\n}","handlingStrategy":"retry","validationCode":"node, err := store.NodeByID(nil, nodeID)\nif err != nil {\n    return err\n}\nif node == nil {\n    return nil\n}","typeGuard":null,"tryCatchPattern":"err := store.UpsertNodeEvents(idx, nodeID, events)\nfor i := 0; i < 3 && err != nil && strings.HasPrefix(err.Error(), \"node update failed\"); i++ {\n    time.Sleep(backoff)\n    err = store.UpsertNodeEvents(idx, nodeID, events)\n}","preventionTips":["Retry transient write failures with backoff at the caller level.","Ensure the memdb transaction is still valid before nested writes.","Monitor server data-dir disk usage; write failures often stem from disk issues.","Escalate to snapshot restore only if write failures persist across restarts."],"tags":["nomad","state-store","storage","node-events","write-failure"],"backgroundTag":"storage-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}