hashicorp/nomad · warning

%w: volume %q could not be found: %v

Error message

%w: volume %q could not be found: %v

What it means

When the CSI plugin's NodeUnstageVolume gRPC call returns a NotFound status, Nomad wraps it with structs.ErrCSIClientRPCIgnorable so callers can detect that this failure can safely be ignored/retried. It means the plugin no longer knows about the volume with that ID — in practice the volume is already unstaged or was removed on the plugin side. The %w wrapping makes errors.Is(err, structs.ErrCSIClientRPCIgnorable) work for callers.

Source

Thrown at plugins/csi/client.go:849

		return fmt.Errorf("missing volumeID")
	}
	if stagingTargetPath == "" {
		return fmt.Errorf("missing stagingTargetPath")
	}

	req := &csipbv1.NodeUnstageVolumeRequest{
		VolumeId:          volumeID,
		StagingTargetPath: stagingTargetPath,
	}

	// NodeUnstageVolume's response contains no extra data. If err == nil, we were
	// successful.
	_, err := c.nodeClient.NodeUnstageVolume(ctx, req, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.NotFound:
			err = fmt.Errorf("%w: volume %q could not be found: %v",
				structs.ErrCSIClientRPCIgnorable, volumeID, err)
		case codes.Internal:
			err = fmt.Errorf("node plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
	}

	return err
}

func (c *client) NodePublishVolume(ctx context.Context, req *NodePublishVolumeRequest, opts ...grpc.CallOption) error {
	if err := c.ensureConnected(ctx); err != nil {
		return err
	}
	if err := req.Validate(); err != nil {
		return fmt.Errorf("validation error: %v", err)
	}

	// NodePublishVolume's response contains no extra data. If err == nil, we were

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Treat as benign: check errors.Is(err, structs.ErrCSIClientRPCIgnorable) and continue with claim cleanup
2. Verify with the plugin that the staging path no longer holds the volume before declaring success
3. If the volume still exists on the backend, re-check the volume ID being passed to unstage
4. Check plugin logs to confirm it dropped the volume record legitimately

Example fix

// before
if err := client.NodeUnstageVolume(ctx, id, path); err != nil {
	return err
}
// after
if err := client.NodeUnstageVolume(ctx, id, path); err != nil {
	if errors.Is(err, structs.ErrCSIClientRPCIgnorable) {
		logger.Warn("volume not found at unstage, treating as unstaged", "error", err)
		return nil
	}
	return err
}

// No local check can fully prevent this: it reflects remote plugin state.
// Optionally verify the volume still exists on the plugin before unstaging:
_, err := client.ControllerValidateCapabilities(ctx, &csi.ControllerValidateVolumeRequest{ExternalID: volumeID, /* ... */})
if err != nil {
	logger.Warn("volume may already be removed", "volume", volumeID)
}

func isCSIIgnorable(err error) bool { return errors.Is(err, structs.ErrCSIClientRPCIgnorable) }

err := client.NodeUnstageVolume(ctx, volumeID, stagingPath)
switch {
case err == nil:
	// unstaged
case errors.Is(err, structs.ErrCSIClientRPCIgnorable):
	// plugin returned NotFound: volume already unstaged; safe to ignore
	logger.Info("volume already unstaged", "volume", volumeID)
default:
	return err
}

When it happens

Trigger: The gRPC plugin returns codes.NotFound during NodeUnstageVolume: the volume was already unstaged (unstage after a previous unstage that actually succeeded), the plugin restarted and lost its volume records, or the volume was deleted from the storage backend while a Nomad claim still referenced it.

Common situations: Node restarts or plugin crashes losing staging state; garbage collection racing with unstage hooks; CSI plugins that report NotFound for unknown volumes on unstage; Nomad attempting unstage after the task's allocation records were already cleaned up.

Related errors

• missing volumeID
• missing stagingTargetPath
• validation error: %v
• nil response from plugin.NodeExpandVolume
• CSI plugin failed to register: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9166c7e715a76154. Report an issue: GitHub.