{"record":{"id":"3a09f226a9d58a49","repo":"hashicorp/nomad","slug":"volume-update-failed-s-v","errorCode":null,"errorMessage":"volume update failed: %s: %v","messagePattern":"volume update failed: (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2933,"sourceCode":"\t\t}\n\t}\n\n\tvolume.ModifyIndex = index\n\tvolume.ModifyTime = now\n\n\t// Allocations are copy on write, so we want to keep the Allocation ID\n\t// but we need to clear the pointer so that we don't store it when we\n\t// write the volume to the state store. We'll get it from the db in\n\t// denormalize.\n\tfor allocID := range volume.ReadAllocs {\n\t\tvolume.ReadAllocs[allocID] = nil\n\t}\n\tfor allocID := range volume.WriteAllocs {\n\t\tvolume.WriteAllocs[allocID] = nil\n\t}\n\n\tif err = txn.Insert(TableCSIVolumes, volume); err != nil {\n\t\treturn fmt.Errorf(\"volume update failed: %s: %v\", id, err)\n\t}\n\n\tif err = txn.Insert(\"index\", &IndexEntry{TableCSIVolumes, index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn txn.Commit()\n}\n\n// CSIVolumeDeregister removes the volume from the server\nfunc (s *StateStore) CSIVolumeDeregister(index uint64, namespace string, ids []string, force bool) error {\n\ttxn := s.db.WriteTxnMsgT(structs.CSIVolumeDeregisterRequestType, index)\n\tdefer txn.Abort()\n\n\tfor _, id := range ids {\n\t\texisting, err := txn.First(TableCSIVolumes, \"id\", namespace, id)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"volume lookup failed: %s: %v\", id, err)","sourceCodeStart":2915,"sourceCodeEnd":2951,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2915-L2951","documentation":"After applying a claim to the in-memory *structs.CSIVolume, the state store re-inserts the row via txn.Insert(TableCSIVolumes, volume). If the memdb insert fails, the store returns this error wrapping the volume ID and underlying cause, and the whole claim transaction is aborted. It indicates a low-level state store write failure rather than a domain-rule rejection.","triggerScenarios":"CSIVolumeClaim commit path failing at txn.Insert for the volume row — typically memdb/BoltDB write errors: disk full, corrupted database file, or an internal transaction state problem.","commonSituations":"Nomad server data_dir disk exhaustion; BoltDB file corruption after unclean shutdown; I/O errors on the host.","solutions":["Check server logs for the wrapped '%v' underlying error to identify the storage cause","Free disk space on the server data_dir and restart the server if BoltDB is reporting I/O errors","Restore the server from a known-good raft snapshot if the state store is corrupt","Retry the claim RPC once storage is healthy"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// No caller-side validation applies; ensure server storage is healthy\n// before issuing writes: check node disk pressure via Nomad/consul metrics","typeGuard":null,"tryCatchPattern":"if err := claimVolume(volID, allocID); err != nil && strings.Contains(err.Error(), \"volume update failed\") {\n\t// transient storage issue: backoff and retry a bounded number of times\n}","preventionTips":["Monitor data_dir disk usage on Nomad servers","Enable alerting on BoltDB write errors","Use clean shutdowns to avoid state corruption","Keep recent raft snapshots for recovery"],"tags":["go","state-store","csi","nomad","storage"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}