hashicorp/nomad · warning

plugin in use

Error message

plugin in use

What it means

ErrCSIPluginInUse is returned when deleting a CSI plugin that still has volumes registered against it. The state store treats this as an error for the Delete operation, but callers like the FSM and CSI endpoint deliberately suppress logging for it because 'plugin in use' is an expected, non-exceptional condition for typical callers.

Source

Thrown at nomad/structs/errors.go:93

	ErrUnknownNode = errors.New(ErrUnknownNodePrefix)

	ErrDeploymentTerminalNoCancel    = errors.New(errDeploymentTerminalNoCancel)
	ErrDeploymentTerminalNoFail      = errors.New(errDeploymentTerminalNoFail)
	ErrDeploymentTerminalNoPause     = errors.New(errDeploymentTerminalNoPause)
	ErrDeploymentTerminalNoPromote   = errors.New(errDeploymentTerminalNoPromote)
	ErrDeploymentTerminalNoResume    = errors.New(errDeploymentTerminalNoResume)
	ErrDeploymentTerminalNoUnblock   = errors.New(errDeploymentTerminalNoUnblock)
	ErrDeploymentTerminalNoRun       = errors.New(errDeploymentTerminalNoRun)
	ErrDeploymentTerminalNoSetHealth = errors.New(errDeploymentTerminalNoSetHealth)
	ErrDeploymentRunningNoUnblock    = errors.New(errDeploymentRunningNoUnblock)

	ErrCSIClientRPCIgnorable  = errors.New("CSI client error (ignorable)")
	ErrCSIClientRPCRetryable  = errors.New("CSI client error (retryable)")
	ErrCSIVolumeMaxClaims     = errors.New("volume max claims reached")
	ErrCSIVolumeUnschedulable = errors.New("volume is currently unschedulable")
	ErrCSIPluginInUse         = errors.New("plugin in use")
)

// IsErrNoLeader returns whether the error is due to there being no leader.
func IsErrNoLeader(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoLeader)
}

// IsErrNoRegionPath returns whether the error is due to there being no path to
// the given region.
func IsErrNoRegionPath(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoRegionPath)
}

// IsErrTokenNotFound returns whether the error is due to the passed token not
// being resolvable.
func IsErrTokenNotFound(err error) bool {
	return err != nil && strings.Contains(err.Error(), errTokenNotFound)
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Delete/deregister all CSI volumes belonging to the plugin first ('nomad volume deregister <id>'), then delete the plugin
2. Stop any jobs still using the plugin's volumes so claims are released
3. Use 'nomad node eligibility' / drain and wait for allocations to move before plugin cleanup
4. Treat the error as an expected outcome in automation: skip or defer deletion when errors.Is(err, structs.ErrCSIPluginInUse)

Example fix

// before
_, _, err := client.NodePlugins().Delete(pluginID, nil)
if err != nil { return err }
// after
_, _, err := client.NodePlugins().Delete(pluginID, nil)
if err != nil && errors.Is(err, structs.ErrCSIPluginInUse) {
    return nil // plugin still has volumes; delete later
}
if err != nil { return err }

plugin, _, err := client.CSIPlugins().Info(pluginID, nil)
if err == nil && plugin != nil && plugin.Volumes != nil {
    for _, vols := range plugin.Volumes {
        if len(vols) > 0 {
            return fmt.Errorf("plugin %s still has %d volumes", pluginID, len(vols))
        }
    }
}

func pluginInUseErr(err error) bool {
    return errors.Is(err, structs.ErrCSIPluginInUse)
}

_, _, err := client.NodePlugins().Delete(pluginID, nil)
if err != nil && errors.Is(err, structs.ErrCSIPluginInUse) {
    log.Printf("plugin %s still in use; deferring delete", pluginID)
    return nil
}
return err

When it happens

Trigger: Calling CSIVolume plugin delete ('nomad plugin delete' / CSIPlugin.Delete RPC, raft CSIPluginDeleteRequestType) while volumes from that plugin still exist in state; also raised by the state store during DeleteCSIPlugin in the FSM.

Common situations: Operator garbage-collects plugins after decommissioning a node but forgot to purge its volumes; deregistering a plugin during cleanup scripts while claims remain; job using the volume is still running.

Related errors

• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required
• CSI.NodeDetachVolume: PluginID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e2afff7fe63488e8. Report an issue: GitHub.