{"record":{"id":"0c63aa2af58c53e2","repo":"hashicorp/nomad","slug":"node-lookup-failed-v","errorCode":null,"errorMessage":"node lookup failed: %v","messagePattern":"node lookup failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1370,"sourceCode":"\tdefer txn.Abort()\n\n\tfor nodeID, events := range nodeEvents {\n\t\tif err := s.upsertNodeEvents(index, nodeID, events, txn); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\treturn txn.Commit()\n}\n\n// upsertNodeEvent upserts a node event for a respective node. It also maintains\n// that a fixed number of node events are ever stored simultaneously, deleting\n// older events once this bound has been reached.\nfunc (s *StateStore) upsertNodeEvents(index uint64, nodeID string, events []*structs.NodeEvent, txn *txn) error {\n\t// Lookup the node\n\texisting, err := txn.First(\"nodes\", \"id\", nodeID)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"node lookup failed: %v\", err)\n\t}\n\tif existing == nil {\n\t\treturn fmt.Errorf(\"node not found\")\n\t}\n\n\t// Copy the existing node\n\texistingNode := existing.(*structs.Node)\n\tcopyNode := existingNode.Copy()\n\tappendNodeEvents(index, copyNode, events)\n\n\t// Insert the node\n\tif err := txn.Insert(\"nodes\", copyNode); err != nil {\n\t\treturn fmt.Errorf(\"node update failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"nodes\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n","sourceCodeStart":1352,"sourceCodeEnd":1388,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1352-L1388","documentation":"upsertNodeEvents wraps a low-level memdb lookup failure when fetching the node whose events are being appended. Unlike 'node not found' (2518), this indicates the txn.First query itself errored, e.g. an internal store/mdb error. Raised through StateStore.UpsertNodeEvents.","triggerScenarios":"txn.First(\"nodes\", \"id\", nodeID) returns a non-nil error — typically an underlying bolt/memdb transaction failure such as a corrupted store, txn misuse, or an I/O error on the backing store.","commonSituations":"State store backend I/O failure (disk full, corruption) during node event updates; internal bugs passing bad arguments to memdb; transaction already aborted when upsertNodeEvents is invoked within a composite write.","solutions":["Inspect the wrapped error (%v) to identify the underlying store failure.","Restart the Nomad server agent to rebuild/reset the state store if corruption is suspected.","Restore the state store from a snapshot (nomad operator snapshot) if the backend is corrupted.","Retry the node event update once the store is healthy."],"exampleFix":"// before\nif err := store.UpsertNodeEvents(idx, nodeID, events); err != nil { return err }\n// after\nif err := store.UpsertNodeEvents(idx, nodeID, events); err != nil {\n    logger.Error(\"node event upsert failed\", \"node\", nodeID, \"err\", err)\n    return err\n}","handlingStrategy":"try-catch","validationCode":"node, err := store.NodeByID(nil, nodeID)\nif err != nil {\n    return fmt.Errorf(\"pre-check lookup failed: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err := store.UpsertNodeEvents(idx, nodeID, events); err != nil {\n    if strings.HasPrefix(err.Error(), \"node lookup failed\") {\n        logger.Error(\"state store lookup failed; check backing store health\", \"err\", err)\n    }\n    return err\n}","preventionTips":["Monitor disk health and space on Nomad server data directories.","Take regular snapshots (nomad operator snapshot) for recovery from corruption.","Alert on any 'lookup failed' state-store errors as infrastructure issues."],"tags":["nomad","state-store","storage","node-events"],"backgroundTag":"storage-lookup-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}