hashicorp/nomad · error

csi_plugins delete error: %v

Error message

csi_plugins delete error: %v

What it means

Wraps a memdb Delete failure when updateOrGCPlugin garbage-collects an empty CSIPlugin (no controllers, nodes, or allocations remain). The row cannot be deleted, failing the enclosing transaction (node upsert/delete, job delete, or alloc terminal update).

Source

Thrown at nomad/state/state_store.go:1573

		err = updateOrGCPlugin(index, txn, plug)
		if err != nil {
			return err
		}
	}

	if err := txn.Insert("index", &IndexEntry{TableCSIPlugins, index}); err != nil {
		return fmt.Errorf("index update failed: %v", err)
	}

	return nil
}

// updateOrGCPlugin updates a plugin but will delete it if the plugin is empty
func updateOrGCPlugin(index uint64, txn Txn, plug *structs.CSIPlugin) error {
	if plug.IsEmpty() {
		err := txn.Delete(TableCSIPlugins, plug)
		if err != nil {
			return fmt.Errorf("csi_plugins delete error: %v", err)
		}
	} else {
		plug.ModifyIndex = index
		err := txn.Insert(TableCSIPlugins, plug)
		if err != nil {
			return fmt.Errorf("csi_plugins update error %s: %v", plug.ID, err)
		}
	}
	return nil
}

// deleteJobFromPlugins removes the allocations of this job from any plugins the job is
// running, possibly deleting the plugin if it's no longer in use. It's called in DeleteJobTxn
func (s *StateStore) deleteJobFromPlugins(index uint64, txn Txn, job *structs.Job) error {
	ws := memdb.NewWatchSet()
	summary, err := s.JobSummaryByID(ws, job.Namespace, job.ID)
	if err != nil {
		return fmt.Errorf("error getting job summary: %v", err)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the triggering operation (job deregistration, node delete, alloc update).
2. Check plugin state via nomad plugin status; force-deregister stale plugins with nomad plugin deregister if rows are stuck.
3. Inspect the wrapped %v error for the memdb root cause.
4. Upgrade Nomad; several CSI GC issues were fixed across patch releases.

Example fix

// before: plugin left stuck, manual cleanup needed
$ nomad plugin status
$ nomad plugin deregister <plugin-id>
// after: ensure CSI volumes/jobs referencing the plugin are purged first
$ nomad volume deregister <volume-id>
$ nomad job stop <csi-job>

// before GC-prone operations, ensure no references remain
plugins, _, _ := client.CSIPlugins().List(nil)
for _, p := range plugins {
    if p.ControllersHealthy == 0 && p.NodesHealthy == 0 {
        client.CSIPlugins().Deregister(p.ID, nil) // explicit, controlled cleanup
    }
}

if err != nil && strings.Contains(err.Error(), "csi_plugins delete error") {
    logger.Error("failed to GC empty CSI plugin; run nomad plugin deregister manually", "err", err)
    return manualPluginCleanup()
}

When it happens

Trigger: Any caller (upsertCSIPluginsForNode, deleteNodeCSIPlugins, deleteJobFromPlugins, updatePluginForTerminalAlloc, updatePluginWithJobSummary) finds plug.IsEmpty() and txn.Delete(TableCSIPlugins, plug) errors.

Common situations: Hit when jobs using CSI volumes complete or nodes deregister and the now-unused plugin should be GC'd; indicates state-store inconsistency or a Nomad CSI bug.

Related errors

• error parsing: root should be an object
• cannot specify Accessor ID
• %w: %v
• failed to read dynamic plugin registry state: %v
• error getting plugin: %s, %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/32180d985fa4887a. Report an issue: GitHub.