hashicorp/nomad · error

volume lookup failed for %s: %v

Error message

volume lookup failed for %s: %v

What it means

Returned by StateStore.CSIVolumeByID when txn.FirstWatch on the csi_volumes "id" index fails. The %s is the volume ID and %v the memdb error. A missing volume does NOT trigger this — CSIVolumeByID returns (nil, nil) in that case — so seeing it always indicates a memdb-level read failure.

Source

Thrown at nomad/state/state_store.go:2725

	iter, err := txn.Get(TableCSIVolumes, "id")
	if err != nil {
		return nil, fmt.Errorf("csi_volumes lookup failed: %v", err)
	}

	ws.Add(iter.WatchCh())

	return iter, nil
}

// CSIVolumeByID is used to lookup a single volume. Returns a copy of the
// volume because its plugins and allocations are denormalized to provide
// accurate Health.
func (s *StateStore) CSIVolumeByID(ws memdb.WatchSet, namespace, id string) (*structs.CSIVolume, error) {
	txn := s.db.ReadTxn()

	watchCh, obj, err := txn.FirstWatch(TableCSIVolumes, "id", namespace, id)
	if err != nil {
		return nil, fmt.Errorf("volume lookup failed for %s: %v", id, err)
	}
	ws.Add(watchCh)

	if obj == nil {
		return nil, nil
	}
	vol := obj.(*structs.CSIVolume)

	// we return the volume with the plugins denormalized by default,
	// because the scheduler needs them for feasibility checking
	return s.csiVolumeDenormalizePluginsTxn(txn, vol.Copy())
}

// CSIVolumesByPluginID looks up csi_volumes by pluginID. Caller should
// snapshot if it wants to also denormalize the plugins.
func (s *StateStore) CSIVolumesByPluginID(ws memdb.WatchSet, namespace, prefix, pluginID string) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped %v for the underlying memdb reason
2. Restart the server so the state store is rebuilt from raft
3. Verify no local patches altered the csi_volumes table indexes
4. File a Nomad issue with nomad operator debug data if it persists

vol, _, err := api.CSIVolumesByID(ws, namespace, id)
if err != nil && strings.Contains(err.Error(), "volume lookup failed for") {
    // memdb read error, not absence: retry or restart server
}

When it happens

Trigger: CSIVolumeByID (backing volume read APIs) hits a FirstWatch error on TableCSIVolumes, i.e. invalid read transaction or schema/index mismatch.

Common situations: Forked/modified state schema; state corruption after a crash; incompatible snapshot restore.

Related errors

• csi_volumes lookup failed: %v
• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ee2edaf13a3fea9a. Report an issue: GitHub.