hashicorp/nomad · error

volume lookup failed: %s %v

Error message

volume lookup failed: %s %v

What it means

Returned by StateStore.CSIVolumesByNodeID while iterating the discovered volume IDs: for each candidate ID a txn.FirstWatch(TableCSIVolumes, "id", namespace, id) is issued, and any failure aborts the whole listing with this message naming the volume ID. The %s identifies which specific volume lookup failed.

Source

Thrown at nomad/state/state_store.go:2840

			continue
		}

		for _, v := range tg.Volumes {
			if v.Type != structs.VolumeTypeCSI {
				continue
			}
			ids[v.Source] = a.Namespace
		}
	}

	// Lookup the raw CSIVolumes to match the other list interfaces
	iter := NewSliceIterator()
	txn := s.db.ReadTxn()
	for id, namespace := range ids {
		if strings.HasPrefix(id, prefix) {
			watchCh, raw, err := txn.FirstWatch(TableCSIVolumes, "id", namespace, id)
			if err != nil {
				return nil, fmt.Errorf("volume lookup failed: %s %v", id, err)
			}
			ws.Add(watchCh)
			iter.Add(raw)
		}
	}

	return iter, nil
}

// CSIVolumesByNamespace looks up the entire csi_volumes table
func (s *StateStore) CSIVolumesByNamespace(ws memdb.WatchSet, namespace, prefix string) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()

	return s.csiVolumesByNamespaceImpl(txn, ws, namespace, prefix)
}

func (s *StateStore) csiVolumesByNamespaceImpl(txn *txn, ws memdb.WatchSet, namespace, prefix string) (memdb.ResultIterator, error) {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Note the volume ID in the message and inspect the wrapped %v for the memdb cause
2. Restart the server to rebuild state from the raft log
3. Check whether any referenced volumes have stale/invalid records after an upgrade and re-register them if needed
4. File a Nomad issue with the debug bundle if persistent

vols, _, err := api.CSIVolumesByNodeID(prefix, nodeID)
if err != nil {
    var volID string
    if _, e := fmt.Sscanf(err.Error(), "volume lookup failed: %s", &volID); e == nil {
        log.Printf("failed reading volume %s: re-register it", volID)
    }
    return err
}

When it happens

Trigger: An alloc on the node references a volume whose per-ID FirstWatch read fails inside memdb — invalid read transaction or index/schema mismatch mid-iteration.

Common situations: State store corruption after crash/upgrade; a fork changed the csi_volumes id index; concurrent memdb failure under memory pressure.

Related errors

• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v
• csi_plugins update error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/dbf8ea39243116af. Report an issue: GitHub.