hashicorp/nomad · error

volume row conversion error: %s

Error message

volume row conversion error: %s

What it means

During CSIVolumeDeregister, the row found for the requested volume cannot be type-asserted to *structs.CSIVolume. This mirrors error 2600 but on the deregister path, and includes the volume ID for diagnosis. It signals internal state corruption or a cross-version row-type mismatch, not a user error.

Source

Thrown at nomad/state/state_store.go:2960

// CSIVolumeDeregister removes the volume from the server
func (s *StateStore) CSIVolumeDeregister(index uint64, namespace string, ids []string, force bool) error {
	txn := s.db.WriteTxnMsgT(structs.CSIVolumeDeregisterRequestType, index)
	defer txn.Abort()

	for _, id := range ids {
		existing, err := txn.First(TableCSIVolumes, "id", namespace, id)
		if err != nil {
			return fmt.Errorf("volume lookup failed: %s: %v", id, err)
		}

		if existing == nil {
			return fmt.Errorf("volume not found: %s", id)
		}

		vol, ok := existing.(*structs.CSIVolume)
		if !ok {
			return fmt.Errorf("volume row conversion error: %s", id)
		}

		// The common case for a volume deregister is when the volume is
		// unused, but we can also let an operator intervene in the case where
		// allocations have been stopped but claims can't be freed because
		// ex. the plugins have all been removed.
		if vol.InUse() {
			if !force || !s.volSafeToForce(txn, vol) {
				return fmt.Errorf("volume in use: %s", id)
			}
		}

		if err = txn.Delete(TableCSIVolumes, existing); err != nil {
			return fmt.Errorf("volume delete failed: %s: %v", id, err)
		}
	}

	if err := txn.Insert("index", &IndexEntry{TableCSIVolumes, index}); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify all servers run the same Nomad version and snapshots match that version
2. Attempt `nomad volume status <id>` to see whether reads also fail
3. If the row is unusable, purge the state store entry by restoring a clean snapshot, or force-delete via operator tooling
4. Report to HashiCorp with logs and snapshot info — internal invariant failure

Example fix

// before: message lacks the offending type
return fmt.Errorf("volume row conversion error: %s", id)
// after
return fmt.Errorf("volume row conversion error: %s: got %T", id, existing)

// Ensure the volume reads cleanly via the API before deregistering
vol, _, err := client.CSIVolumes().Get(volumeID, nil)
if err != nil || vol == nil {
	return fmt.Errorf("volume %s unreadable: %w", volumeID, err)
}

func asCSIVolume(row interface{}) (*structs.CSIVolume, bool) {
	v, ok := row.(*structs.CSIVolume)
	return v, ok
}

if err := deregisterVolume(volID); err != nil && strings.Contains(err.Error(), "volume row conversion error") {
	// internal corruption: escalate to operator, do not retry
}

When it happens

Trigger: nomad volume deregister against a volume table row stored with an unexpected type — corrupted state store, snapshot from an incompatible Nomad version, or patched builds inserting foreign structs.

Common situations: Cross-version raft snapshot restore; manual state tampering; storage corruption after crash.

Related errors

• volume row conversion error
• structs.ErrUnknownAllocationPrefix
• volume update failed: %s: %v
• volume delete failed: %s: %v
• error parsing: root should be an object

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/42ef8ffddd9de9d4. Report an issue: GitHub.