hashicorp/nomad · error

alloc lookup failed: %v

Error message

alloc lookup failed: %v

What it means

Returned by StateStore.CSIVolumesByNodeID when the delegated call to s.AllocsByNode(ws, nodeID) fails while collecting allocations for a node to discover which CSI volumes are in use. The underlying alloc lookup error is preserved in %v.

Source

Thrown at nomad/state/state_store.go:2811

	// Filter the iterator by ID prefix
	f := func(raw any) bool {
		v, ok := raw.(*structs.CSIVolume)
		if !ok {
			return false
		}
		return !strings.HasPrefix(v.ID, prefix)
	}
	wrap := memdb.NewFilterIterator(iter, f)
	return wrap, nil
}

// CSIVolumesByNodeID looks up CSIVolumes in use on a node. Caller should
// snapshot if it wants to also denormalize the plugins.
func (s *StateStore) CSIVolumesByNodeID(ws memdb.WatchSet, prefix, nodeID string) (memdb.ResultIterator, error) {
	allocs, err := s.AllocsByNode(ws, nodeID)
	if err != nil {
		return nil, fmt.Errorf("alloc lookup failed: %v", err)
	}

	// Find volume ids for CSI volumes in running allocs, or allocs that we desire to run
	ids := map[string]string{} // Map volumeID to Namespace
	for _, a := range allocs {
		tg := a.Job.LookupTaskGroup(a.TaskGroup)

		if !(a.DesiredStatus == structs.AllocDesiredStatusRun ||
			a.ClientStatus == structs.AllocClientStatusRunning) ||
			len(tg.Volumes) == 0 {
			continue
		}

		for _, v := range tg.Volumes {
			if v.Type != structs.VolumeTypeCSI {
				continue
			}
			ids[v.Source] = a.Namespace

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the wrapped %v for the root memdb error from the allocs table
2. Restart the server to rebuild state from raft
3. Verify consistent Nomad versions across all server nodes
4. Capture a debug bundle and escalate to Nomad if it recurs

vols, _, err := api.CSIVolumesByNodeID(prefix, nodeID)
if err != nil && strings.Contains(err.Error(), "alloc lookup failed") {
    // allocs-table read failure: backoff-retry, then restart server if persistent
    return retryWithBackoff(func() error { _, _, e := api.CSIVolumesByNodeID(prefix, nodeID); return e })
}

When it happens

Trigger: Listing CSI volumes by node ID when AllocsByNode's memdb Get on the allocs table ("node" index) returns an error.

Common situations: Corrupt allocs index after a bad upgrade or forked schema; OOM-induced memdb failure; usually seen together with other state-store read errors.

Related errors

• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v
• csi_plugins update error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7be41673cb3baa25. Report an issue: GitHub.