hashicorp/nomad · warning

%w: volume %q or node %q could not be found: %v

Error message

%w: volume %q or node %q could not be found: %v

What it means

Raised in ControllerUnpublishVolume when the controller plugin returns gRPC NotFound. Nomad comments that the volume and node were validated to exist, so a NotFound here typically means the volume was previously checkpointed as published but no longer exists at the storage backend. Nomad wraps the error with structs.ErrCSIClientRPCIgnorable so callers can detect that it is safe to ignore (e.g. log for diagnostics and treat the unpublish as complete) via errors.Is.

Source

Thrown at plugins/csi/client.go:360

	if err := c.ensureConnected(ctx); err != nil {
		return nil, err
	}
	err := req.Validate()
	if err != nil {
		return nil, err
	}

	upbrequest := req.ToCSIRepresentation()
	_, err = c.controllerClient.ControllerUnpublishVolume(ctx, upbrequest, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.NotFound:
			// we'll have validated the volume and node *should* exist at the
			// server, so if we get a not-found here it's because we've previously
			// checkpointed. we'll return an error so the caller can log it for
			// diagnostic purposes.
			err = fmt.Errorf("%w: volume %q or node %q could not be found: %v",
				structs.ErrCSIClientRPCIgnorable, req.ExternalID, req.NodeID, err)
		case codes.Internal:
			err = fmt.Errorf("controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return nil, err
	}

	return &ControllerUnpublishVolumeResponse{}, nil
}

func (c *client) ControllerValidateCapabilities(ctx context.Context, req *ControllerValidateVolumeRequest, opts ...grpc.CallOption) error {
	if err := c.ensureConnected(ctx); err != nil {
		return err
	}
	if req.ExternalID == "" {
		return fmt.Errorf("missing volume ID")
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Treat it as ignorable: check errors.Is(err, structs.ErrCSIClientRPCIgnorable) and log the volume/node IDs for diagnostics instead of failing the unpublish flow.
2. Clean up Nomad's view: deregister or GC the volume (nomad volume deregister <id>) so it stops being checkpointed as published.
3. If the volume should exist, verify its ID against the storage provider and re-register/recreate it before unpublishing.
4. Audit for out-of-band deletions (CI scripts, consoles) that bypass Nomad volume lifecycle.

Example fix

// before
err := csi.ControllerUnpublishVolume(ctx, req)
if err != nil {
    return err // blocks claim GC on already-deleted volumes
}
// after
err := csi.ControllerUnpublishVolume(ctx, req)
if err != nil {
    if errors.Is(err, structs.ErrCSIClientRPCIgnorable) {
        logger.Warn("unpublish not found; ignoring", "volume", req.ExternalID)
        return nil
    }
    return err
}

// Before unpublishing, confirm the volume still exists and which nodes claim it.
vol, _, err := nomadClient.CSIVolumes().Get(nil, req.ExternalID)
if err != nil {
    if err is 404: // already deregistered at Nomad; skip unpublish
        return nil
    return err
}
// avoid unpublishing for checkpointed volumes that no longer exist at the backend

// Nomad marks this case with a wrapped sentinel:
func isIgnorableUnpublishNotFound(err error) bool {
    return errors.Is(err, structs.ErrCSIClientRPCIgnorable)
}

_, err := csi.ControllerUnpublishVolume(ctx, req)
switch {
case err == nil:
    return nil
case errors.Is(err, structs.ErrCSIClientRPCIgnorable):
    logger.Warn("volume/node not found at backend; ignoring", "volume", req.ExternalID, "node", req.NodeID)
    return nil
default:
    return err
}

When it happens

Trigger: c.ControllerUnpublishVolume() called for req.ExternalID/req.NodeID and the plugin replies codes.NotFound — the volume was deleted out-of-band at the provider, or the checkpointed attachment no longer exists server-side.

Common situations: An operator deleted the volume at the cloud console while Nomad still tracked it; a leaked claim from a node that was destroyed; GC removed the volume while an unpublish was still pending in the checkpoint state.

Related errors

• volume %q could not be found: %v
• volume %q content source %q does not exist: %v
• %w: volume %q could not be found: %v
• nil response from plugin.NodeExpandVolume
• CSI plugin failed to register: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/98ab5c90a8f6c367. Report an issue: GitHub.