What causes "volume delete failed: %s: %v" in hashicorp/nomad?

Inspect the wrapped '%v' underlying cause in server logs

volume delete failed: %s: %v

Error message

volume delete failed: %s: %v

What it means

After the in-use check passes, CSIVolumeDeregister deletes the row with txn.Delete(TableCSIVolumes, existing). If that memdb delete fails, this error wraps the volume ID and cause and aborts the transaction. It is a low-level storage failure, not a domain rule.

Source

Thrown at nomad/state/state_store.go:2974

		}

		vol, ok := existing.(*structs.CSIVolume)
		if !ok {
			return fmt.Errorf("volume row conversion error: %s", id)
		}

		// The common case for a volume deregister is when the volume is
		// unused, but we can also let an operator intervene in the case where
		// allocations have been stopped but claims can't be freed because
		// ex. the plugins have all been removed.
		if vol.InUse() {
			if !force || !s.volSafeToForce(txn, vol) {
				return fmt.Errorf("volume in use: %s", id)
			}
		}

		if err = txn.Delete(TableCSIVolumes, existing); err != nil {
			return fmt.Errorf("volume delete failed: %s: %v", id, err)
		}
	}

	if err := txn.Insert("index", &IndexEntry{TableCSIVolumes, index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return txn.Commit()
}

// volSafeToForce checks if the any of the remaining allocations
// are in a non-terminal state.
func (s *StateStore) volSafeToForce(txn Txn, v *structs.CSIVolume) bool {
	v = v.Copy()
	vol, err := s.csiVolumeDenormalizeTxn(txn, nil, v)
	if err != nil {
		return false
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped '%v' underlying cause in server logs
Free disk space / fix I/O issues on the server data_dir
Restart the server; restore from a known-good snapshot if corruption persists
Re-run the deregister once the state store is healthy

Defensive patterns

Strategy: retry

Validate before calling

// Ensure server health before issuing deletes
_, _, err := client.Status().Leader()
if err != nil { return fmt.Errorf("cluster unhealthy: %w", err) }

Try / catch

if err := deregisterVolume(volID); err != nil && strings.Contains(err.Error(), "volume delete failed") {
	// storage failure: fix disk/state, then retry deregistration
}

Prevention

Monitor data_dir disk usage and I/O health
Use clean server shutdowns to protect BoltDB
Keep regular raft snapshots
Re-run deregistration after storage recovery

When it happens

Trigger: Volume deregister where the memdb/BoltDB delete of the volume row errors — disk full, I/O failure, or corrupted state store.

Common situations: Disk exhaustion on the Nomad server data_dir; BoltDB corruption after unclean shutdown; underlying storage hardware failure.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/27eb4f55095e1b1b. Report an issue: GitHub.