hashicorp/nomad · error

CSI.NodeDetachVolume: %v

Error message

CSI.NodeDetachVolume: %v

What it means

NodeDetachVolume is the client RPC endpoint that asks a CSI node plugin to unmount/detach a volume from an allocation. This error wraps a failure from csimanager.ManagerForPlugin, meaning the client could not locate or initialize a manager for the requested CSI plugin (req.PluginID). It aborts the detach before any unmount is attempted.

Source

Thrown at client/csi_endpoint.go:521

	// The following block of validation checks should not be reached on a
	// real Nomad cluster. They serve as a defensive check before forwarding
	// requests to plugins, and to aid with development.
	if req.PluginID == "" {
		return errors.New("CSI.NodeDetachVolume: PluginID is required")
	}
	if req.VolumeID == "" {
		return errors.New("CSI.NodeDetachVolume: VolumeID is required")
	}
	if req.AllocID == "" {
		return errors.New("CSI.NodeDetachVolume: AllocID is required")
	}

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	manager, err := c.c.csimanager.ManagerForPlugin(ctx, req.PluginID)
	if err != nil {
		return fmt.Errorf("CSI.NodeDetachVolume: %v", err)
	}

	usageOpts := &csimanager.UsageOptions{
		ReadOnly:       req.ReadOnly,
		AttachmentMode: req.AttachmentMode,
		AccessMode:     req.AccessMode,
	}

	err = manager.UnmountVolume(ctx, req.VolumeNamespace, req.VolumeID, req.ExternalID, req.AllocID, usageOpts)
	if err != nil && !errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {
		// if the unmounting previously happened but the server failed to
		// checkpoint, we'll get an error from Unmount but can safely
		// ignore it.
		return fmt.Errorf("CSI.NodeDetachVolume: %v", err)
	}
	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the plugin ID in the detach request matches a running CSI plugin on the target node (nomad plugin status).
2. Restart or redeploy the CSI plugin task on the node so a manager can be created.
3. Check nomad agent logs on the client for the underlying ManagerForPlugin error to see whether the plugin failed to load or was deregistered.
4. Re-run the detach once the plugin is healthy; if the allocation is gone, the claim cleanup can be done from the server side.

Example fix

// before: detaching with stale plugin id
client.CSIVolumes().Detach(ctx, &cstructs.ClientCSIDetachRequest{PluginID: "old-plugin", ...})
// after: look up the live plugin first
pluginID := "registry.aws.ebs" // from `nomad plugin status`
client.CSIVolumes().Detach(ctx, &cstructs.ClientCSIDetachRequest{PluginID: pluginID, ...})

// confirm plugin exists before detaching
// nomad plugin status <plugin-id>  -> must show node plugins
if pluginStatus == nil || len(pluginStatus.NodePlugins) == 0 {
  return fmt.Errorf("plugin %q not running on node", pluginID)
}

func hasCSIPlugin(plugins []*api.Plugin, id string) bool {
  for _, p := range plugins { if p.ID == id { return true } }
  return false
}

err := volumes.Detach(ctx, volID, nodeID, pluginID)
if err != nil && strings.Contains(err.Error(), "CSI.NodeDetachVolume") {
  // plugin unavailable: re-check plugin status and retry after plugin restart
  log.Warnf("csi plugin %s unavailable: %v", pluginID, err)
}

When it happens

Trigger: An API caller (nomad volume detach / allocation stop) sends NodeDetachVolume with a PluginID for which no plugin instance exists on the client, the plugin is not running, or the plugin manager cannot be created for that plugin type.

Common situations: Volume deregistered or plugin upgraded/restarted on the node so the PluginID no longer matches a live plugin; typo'd or stale plugin ID after node restart; CSI plugin crashed so the manager for that plugin is unavailable.

Related errors

• CSI Plugin loaded incorrectly
• CSI.ControllerListVolumes: plugin returned an invalid entry
• controller attach volume: %v
• controller validate volume: %v
• controller detach volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/95a89dbd68c97ffd. Report an issue: GitHub.