hashicorp/nomad · error

volume insert: %v

Error message

volume insert: %v

What it means

Wrapped error returned by StateStore.upsertCSIVolume when the underlying memdb transaction fails to insert a structs.CSIVolume row into the csi_volumes table. The %v carries the raw go-memdb insert error, which in practice is almost always a memdb object-type mismatch or an out-of-txn write attempt. It aborts the whole state-store transaction, so the volume registration is not persisted.

Source

Thrown at nomad/state/state_store.go:2690

		} else {
			v.CreateIndex = index
		}
		v.ModifyIndex = index

		// Allocations are copy on write, so we want to keep the Allocation ID
		// but we need to clear the pointer so that we don't store it when we
		// write the volume to the state store. We'll get it from the db in
		// denormalize.
		for allocID := range v.ReadAllocs {
			v.ReadAllocs[allocID] = nil
		}
		for allocID := range v.WriteAllocs {
			v.WriteAllocs[allocID] = nil
		}

		err = txn.Insert(TableCSIVolumes, v)
		if err != nil {
			return fmt.Errorf("volume insert: %v", err)
		}
	}

	if err := txn.Insert("index", &IndexEntry{TableCSIVolumes, index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return txn.Commit()
}

// CSIVolumes returns the unfiltered list of all volumes. Caller should
// snapshot if it wants to also denormalize the plugins.
func (s *StateStore) CSIVolumes(ws memdb.WatchSet) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()
	defer txn.Abort()

	iter, err := txn.Get(TableCSIVolumes, "id")
	if err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the Nomad server logs for the underlying %v message to identify the exact memdb error
2. Verify all servers run the same Nomad version and no schema-modifying forks are in use
3. If it appeared after an upgrade/restore, re-run nomad operator snapshot save/restore with a matching version
4. Restart the server to rebuild the in-memory state store from the raft log; report to Nomad if it persists

Example fix

// before (custom code inserting wrong type)
txn.Insert(TableCSIVolumes, rawRow)
// after
txn.Insert(TableCSIVolumes, v.(*structs.CSIVolume))

// ensure the object is the correct type before upsert
if _, ok := vol.(*structs.CSIVolume); !ok {
    return fmt.Errorf("not a CSIVolume")
}

func isCSIVolume(v any) bool { _, ok := v.(*structs.CSIVolume); return ok }

if err := api.CSIVolumeRegister(vol); err != nil {
    if strings.Contains(err.Error(), "volume insert:") {
        // memdb insert failure: restart server / re-register
    }
    return err
}

When it happens

Trigger: txn.Insert(TableCSIVolumes, v) returns an error: the object being inserted is not a *structs.CSIVolume registered with the table schema, or the write transaction was already invalidated/aborted by a prior failing operation in the same UpsertCSIVolume call.

Common situations: Internal state corruption after a Nomad upgrade where the CSIVolume struct changed schema; snapshots restored from incompatible versions; custom forks patching the memdb table schema; raft log replay hitting a malformed CSIVolumeRegistration.

Related errors

• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v
• csi_plugins update error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d64f74753422bff1. Report an issue: GitHub.