hashicorp/nomad · info

%w: %v

Error message

%w: %v

What it means

During unpublishVolume, if the volume's host target path no longer exists on disk AND the controller unmount RPC failed with a "no mount point" error, the rpcErr is wrapped with structs.ErrCSIClientRPCIgnorable. This marks the failure as benign — a previous GC attempt already destroyed the volume on the node even though its controller RPCs failed — so callers (GC) can ignore it.

Source

Thrown at client/pluginmanager/csimanager/volume.go:298

	logger := hclog.FromContext(ctx)
	logger.Trace("unpublishing volume", "plugin_target_path", pluginTargetPath)

	// CSI NodeUnpublishVolume errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	rpcErr := v.plugin.NodeUnpublishVolume(ctx, remoteID, pluginTargetPath,
		grpc_retry.WithPerRetryTimeout(DefaultMountActionTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)),
	)

	hostTargetPath := v.targetForVolume(v.mountRoot, volID, allocID, usage)
	if _, err := os.Stat(hostTargetPath); os.IsNotExist(err) {
		if rpcErr != nil && strings.Contains(rpcErr.Error(), "no mount point") {
			// host target path was already destroyed, nothing to do here.
			// this helps us in the case that a previous GC attempt cleaned
			// up the volume on the node but the controller RPCs failed
			rpcErr = fmt.Errorf("%w: %v", structs.ErrCSIClientRPCIgnorable, rpcErr)
		}
		return rpcErr
	}

	logger.Trace("removing host path", "host_target_path", hostTargetPath)

	// Host Target Path was not cleaned up, attempt to do so here. If it's still
	// a mount then removing the dir will fail and we'll return any rpcErr and the
	// file error.
	rmErr := os.Remove(hostTargetPath)
	if rmErr != nil {
		return combineErrors(rpcErr, rmErr)
	}

	// We successfully removed the directory, return any rpcErrors that were
	// encountered, but because we got here, they were probably flaky or was
	// cleaned up externally.
	return fmt.Errorf("%w: %v", structs.ErrCSIClientRPCIgnorable, rpcErr)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Nothing to fix in most cases: treat errors wrapping ErrCSIClientRPCIgnorable as safe to skip during GC
2. If this appears repeatedly, investigate why prior unmounts left the controller RPC failing (stale attachments at the storage backend)
3. Verify the volume is fully detached at the storage provider and clean up leftover controller attachments manually
4. Check for a plugin version that mishandles unpublish of already-removed mounts and upgrade the CSI plugin

Example fix

// before: treating every unpublish error as fatal
if err := client.CSI().UnpublishVolume(...); err != nil {
    return err
}

// after: ignore ignorable unpublish errors during GC
if err := client.CSI().UnpublishVolume(...); err != nil {
    if errors.Is(err, structs.ErrCSIClientRPCIgnorable) {
        logger.Warn("ignorable CSI unpublish error, volume already destroyed", "err", err)
        return nil
    }
    return err
}

// Nothing to check before the API; the error only occurs after prior GC already removed the mount.
null

err := unpublishVolume(ctx, volID, allocID, usage)
if err != nil && errors.Is(err, structs.ErrCSIClientRPCIgnorable) {
	// host path already destroyed by a previous GC; safe to proceed
	logger.Warn("volume already unpublished on node; ignoring", "vol", volID)
	err = nil
}
return err

When it happens

Trigger: Node-local unmount cleanup runs while the host target path is already gone (prior GC removed it) and the preceding controller UnpublishVolume RPC returned an error mentioning "no mount point".

Common situations: Repeated GC cycles on the same allocation; a controller that reports 'no mount point' after the node already detached; retrying volume unpublish after a partially completed earlier attempt.

Related errors

• missing policy name
• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8da384faad2af6d6. Report an issue: GitHub.