hashicorp/nomad · error

volume row conversion error

Error message

volume row conversion error

What it means

Returned by Nomad's state store when a CSI volume claim update (CSIVolumeClaim) finds a row in the CSIVolumes table that cannot be type-asserted to *structs.CSIVolume. The memdb row store returned an object of an unexpected type, so the store refuses to mutate it rather than panic. This indicates internal state corruption or a schema/version mismatch, not a user API misuse.

Source

Thrown at nomad/state/state_store.go:2884

	return iter, nil
}

// CSIVolumeClaim updates the volume's claim count and allocation list
func (s *StateStore) CSIVolumeClaim(index uint64, now int64, namespace, id string, claim *structs.CSIVolumeClaim) error {
	txn := s.db.WriteTxnMsgT(structs.CSIVolumeClaimRequestType, index)
	defer txn.Abort()

	row, err := txn.First(TableCSIVolumes, "id", namespace, id)
	if err != nil {
		return fmt.Errorf("volume lookup failed: %s: %v", id, err)
	}
	if row == nil {
		return fmt.Errorf("volume not found: %s", id)
	}

	orig, ok := row.(*structs.CSIVolume)
	if !ok {
		return fmt.Errorf("volume row conversion error")
	}

	var alloc *structs.Allocation
	if claim.State == structs.CSIVolumeClaimStateTaken {
		alloc, err = s.allocByIDImpl(txn, nil, claim.AllocationID)
		if err != nil {
			s.logger.Error("AllocByID failed", "error", err)
			return fmt.Errorf(structs.ErrUnknownAllocationPrefix)
		}
		if alloc == nil {
			s.logger.Error("AllocByID failed to find alloc", "alloc_id", claim.AllocationID)
		}
	}

	volume, err := s.csiVolumeDenormalizePluginsTxn(txn, orig.Copy())
	if err != nil {
		return err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the Nomad server version consistency across the cluster and ensure the snapshot/raft data matches that version
2. Inspect the state store entry for the volume via `nomad volume status <id>` and server logs to confirm the row is corrupt
3. Delete and re-register the affected CSI volume (nomad volume deregister, then nomad volume register)
4. File an issue with HashiCorp with the server log and snapshot details — this is an internal invariant failure

Example fix

// Not caller-fixable; state-store-side hardening
// before
orig, ok := row.(*structs.CSIVolume)
if !ok {
	return fmt.Errorf("volume row conversion error")
}
// after
orig, ok := row.(*structs.CSIVolume)
if !ok {
	return fmt.Errorf("volume row conversion error: got %T", row)
}

// Go caller via API: pre-check the volume exists and reads cleanly
_, err := client.CSIVolumes().Get(volumeID)
if err != nil {
	return fmt.Errorf("volume %s unreadable, skipping claim: %w", volumeID, err)
}

func asCSIVolume(row interface{}) (*structs.CSIVolume, bool) {
	v, ok := row.(*structs.CSIVolume)
	return v, ok
}

err := client.CSIVolumes().Claim(claim)
var apiErr *api.QueryError
if errors.As(err, &apiErr) && strings.Contains(err.Error(), "conversion error") {
	// internal corruption: escalate to operator, do not retry blindly
}

When it happens

Trigger: Calling state store CSIVolumeClaim (via volume claim RPC) when the row stored under TableCSIVolumes for the requested namespace/id is not a *structs.CSIVolume — typically after a corrupted or hand-edited BoltDB/memdb snapshot, a downgrade across Nomad versions that changed the stored row type, or a code bug inserting a different struct under the volume table.

Common situations: Restoring a raft snapshot from a different Nomad version; mixing state_store data across major upgrades; custom patched Nomad builds writing foreign row types; corrupted state store after crash.

Related errors

• structs.ErrUnknownAllocationPrefix
• volume update failed: %s: %v
• volume row conversion error: %s
• volume delete failed: %s: %v
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d557c121dbb626d4. Report an issue: GitHub.