{"record":{"id":"d64f74753422bff1","repo":"hashicorp/nomad","slug":"volume-insert-v","errorCode":null,"errorMessage":"volume insert: %v","messagePattern":"volume insert: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2690,"sourceCode":"\t\t} else {\n\t\t\tv.CreateIndex = index\n\t\t}\n\t\tv.ModifyIndex = index\n\n\t\t// Allocations are copy on write, so we want to keep the Allocation ID\n\t\t// but we need to clear the pointer so that we don't store it when we\n\t\t// write the volume to the state store. We'll get it from the db in\n\t\t// denormalize.\n\t\tfor allocID := range v.ReadAllocs {\n\t\t\tv.ReadAllocs[allocID] = nil\n\t\t}\n\t\tfor allocID := range v.WriteAllocs {\n\t\t\tv.WriteAllocs[allocID] = nil\n\t\t}\n\n\t\terr = txn.Insert(TableCSIVolumes, v)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"volume insert: %v\", err)\n\t\t}\n\t}\n\n\tif err := txn.Insert(\"index\", &IndexEntry{TableCSIVolumes, index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn txn.Commit()\n}\n\n// CSIVolumes returns the unfiltered list of all volumes. Caller should\n// snapshot if it wants to also denormalize the plugins.\nfunc (s *StateStore) CSIVolumes(ws memdb.WatchSet) (memdb.ResultIterator, error) {\n\ttxn := s.db.ReadTxn()\n\tdefer txn.Abort()\n\n\titer, err := txn.Get(TableCSIVolumes, \"id\")\n\tif err != nil {","sourceCodeStart":2672,"sourceCodeEnd":2708,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2672-L2708","documentation":"Wrapped error returned by StateStore.upsertCSIVolume when the underlying memdb transaction fails to insert a structs.CSIVolume row into the csi_volumes table. The %v carries the raw go-memdb insert error, which in practice is almost always a memdb object-type mismatch or an out-of-txn write attempt. It aborts the whole state-store transaction, so the volume registration is not persisted.","triggerScenarios":"txn.Insert(TableCSIVolumes, v) returns an error: the object being inserted is not a *structs.CSIVolume registered with the table schema, or the write transaction was already invalidated/aborted by a prior failing operation in the same UpsertCSIVolume call.","commonSituations":"Internal state corruption after a Nomad upgrade where the CSIVolume struct changed schema; snapshots restored from incompatible versions; custom forks patching the memdb table schema; raft log replay hitting a malformed CSIVolumeRegistration.","solutions":["Check the Nomad server logs for the underlying %v message to identify the exact memdb error","Verify all servers run the same Nomad version and no schema-modifying forks are in use","If it appeared after an upgrade/restore, re-run nomad operator snapshot save/restore with a matching version","Restart the server to rebuild the in-memory state store from the raft log; report to Nomad if it persists"],"exampleFix":"// before (custom code inserting wrong type)\ntxn.Insert(TableCSIVolumes, rawRow)\n// after\ntxn.Insert(TableCSIVolumes, v.(*structs.CSIVolume))","handlingStrategy":"try-catch","validationCode":"// ensure the object is the correct type before upsert\nif _, ok := vol.(*structs.CSIVolume); !ok {\n    return fmt.Errorf(\"not a CSIVolume\")\n}","typeGuard":"func isCSIVolume(v any) bool { _, ok := v.(*structs.CSIVolume); return ok }","tryCatchPattern":"if err := api.CSIVolumeRegister(vol); err != nil {\n    if strings.Contains(err.Error(), \"volume insert:\") {\n        // memdb insert failure: restart server / re-register\n    }\n    return err\n}","preventionTips":["Keep all Nomad servers on the same version","Never patch memdb table schemas in forks","Restore snapshots only into matching Nomad versions","Monitor server memory to avoid OOM-induced memdb failures"],"tags":["state-store","memdb","csi","nomad"],"backgroundTag":"memdb-insert-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}