hashicorp/nomad · error

volume lookup failed: %s: %v

Error message

volume lookup failed: %s: %v

What it means

Returned by StateStore.CSIVolumeClaim when txn.First(TableCSIVolumes, "id", namespace, id) errors while locating the volume to attach/detach a claim (used by the CSI volume claim RPC during alloc setup/teardown). The %s is the volume ID and %v the memdb error. A nil row is handled separately (error 2599), so this error is strictly a read failure.

Source

Thrown at nomad/state/state_store.go:2876

	iter, err := txn.Get(TableCSIVolumes, "id_prefix", namespace, prefix)
	if err != nil {
		return nil, fmt.Errorf("volume lookup failed: %v", err)
	}

	ws.Add(iter.WatchCh())

	return iter, nil
}

// CSIVolumeClaim updates the volume's claim count and allocation list
func (s *StateStore) CSIVolumeClaim(index uint64, now int64, namespace, id string, claim *structs.CSIVolumeClaim) error {
	txn := s.db.WriteTxnMsgT(structs.CSIVolumeClaimRequestType, index)
	defer txn.Abort()

	row, err := txn.First(TableCSIVolumes, "id", namespace, id)
	if err != nil {
		return fmt.Errorf("volume lookup failed: %s: %v", id, err)
	}
	if row == nil {
		return fmt.Errorf("volume not found: %s", id)
	}

	orig, ok := row.(*structs.CSIVolume)
	if !ok {
		return fmt.Errorf("volume row conversion error")
	}

	var alloc *structs.Allocation
	if claim.State == structs.CSIVolumeClaimStateTaken {
		alloc, err = s.allocByIDImpl(txn, nil, claim.AllocationID)
		if err != nil {
			s.logger.Error("AllocByID failed", "error", err)
			return fmt.Errorf(structs.ErrUnknownAllocationPrefix)
		}
		if alloc == nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v message in the server logs
2. Restart the server to rebuild state from the raft log and retry the claim
3. Re-run the affected allocation (nomad alloc stop / redeploy) once the store is healthy
4. Escalate to Nomad with a debug bundle if reproducible

// check the volume exists before submitting a claim
vol, _, err := api.CSIVolumesByID(ws, namespace, volumeID)
if err != nil {
    return fmt.Errorf("state store read error, retry later: %w", err)
}
if vol == nil {
    return fmt.Errorf("volume %s does not exist in namespace %s", volumeID, namespace)
}

err := api.CSIVolumeClaim(&structs.CSIVolumeClaimRequest{VolumeID: id, ...})
if err != nil && strings.Contains(err.Error(), "volume lookup failed") {
    // memdb read failure: backoff-retry, then restart server
    return retryWithBackoff(func() error { return api.CSIVolumeClaim(req) })
}

When it happens

Trigger: Client submits CSIVolumeClaimRequest (claim/unclaim on alloc run or stop) and the First lookup on the csi_volumes id index fails.

Common situations: State store corruption after crash/restore; fork-modified schema; OOM pressure on the leader.

Related errors

• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v
• csi_plugins update error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f3665b803c3849c2. Report an issue: GitHub.