hashicorp/nomad · error

volume lookup failed: %v

Error message

volume lookup failed: %v

What it means

Returned by StateStore.CSIVolumesByPluginID when txn.Get(TableCSIVolumes, "plugin_id", pluginID) fails to build the iterator over volumes registered with a given CSI plugin. Note the namespace filter is applied afterwards via a SliceIterator filter, so this error is purely the memdb Get on the plugin_id index failing.

Source

Thrown at nomad/state/state_store.go:2746

	if obj == nil {
		return nil, nil
	}
	vol := obj.(*structs.CSIVolume)

	// we return the volume with the plugins denormalized by default,
	// because the scheduler needs them for feasibility checking
	return s.csiVolumeDenormalizePluginsTxn(txn, vol.Copy())
}

// CSIVolumesByPluginID looks up csi_volumes by pluginID. Caller should
// snapshot if it wants to also denormalize the plugins.
func (s *StateStore) CSIVolumesByPluginID(ws memdb.WatchSet, namespace, prefix, pluginID string) (memdb.ResultIterator, error) {
	txn := s.db.ReadTxn()

	iter, err := txn.Get(TableCSIVolumes, "plugin_id", pluginID)
	if err != nil {
		return nil, fmt.Errorf("volume lookup failed: %v", err)
	}

	// Filter the iterator by namespace
	f := func(raw any) bool {
		v, ok := raw.(*structs.CSIVolume)
		if !ok {
			return false
		}
		return v.Namespace != namespace && strings.HasPrefix(v.ID, prefix)
	}

	wrap := memdb.NewFilterIterator(iter, f)
	return wrap, nil
}

// CSIVolumesByIDPrefix supports search. Caller should snapshot if it wants to
// also denormalize the plugins. If using a prefix with the wildcard namespace,
// the results will not use the index prefix.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v message in the server log
2. Restart the agent to rebuild state from the raft log
3. Ensure all servers run an unmodified, matching Nomad version
4. Report with debug bundle if reproducible

iter, _, err := api.CSIVolumesByPluginID(prefix, pluginID)
if err != nil && strings.Contains(err.Error(), "volume lookup failed") {
    // plugin_id index Get failed: retry after server restart
}

When it happens

Trigger: Listing volumes by plugin ID (e.g. node/plugin status endpoints) while the "plugin_id" index Get errors — read-txn invalidity or schema mismatch.

Common situations: Custom builds with altered indexes; corrupted in-memory store after a bug or OOM; upgrade path issues.

Related errors

• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v
• csi_plugins update error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1dc15df64d35525e. Report an issue: GitHub.