hashicorp/nomad · error

volume %q could not be found: %v

Error message

volume %q could not be found: %v

What it means

The CSI controller plugin responded to ValidateVolumeCapabilities with gRPC code NotFound, meaning the volume identified by ExternalID does not exist on the storage backend. The client rewrites the gRPC status into this descriptive error including the volume ID and the original gRPC error text, since the plugin is the authority on whether a volume exists.

Source

Thrown at plugins/csi/client.go:389

func (c *client) ControllerValidateCapabilities(ctx context.Context, req *ControllerValidateVolumeRequest, opts ...grpc.CallOption) error {
	if err := c.ensureConnected(ctx); err != nil {
		return err
	}
	if req.ExternalID == "" {
		return fmt.Errorf("missing volume ID")
	}

	if req.Capabilities == nil {
		return fmt.Errorf("missing Capabilities")
	}

	creq := req.ToCSIRepresentation()
	resp, err := c.controllerClient.ValidateVolumeCapabilities(ctx, creq, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.NotFound:
			err = fmt.Errorf("volume %q could not be found: %v", req.ExternalID, err)
		case codes.Internal:
			err = fmt.Errorf("controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return err
	}

	if resp.Message != "" {
		// this should only ever be set if Confirmed isn't set, but
		// it's not a validation failure.
		c.logger.Debug(resp.Message)
	}

	// The protobuf accessors below safely handle nil pointers.
	// The CSI spec says we can only assert the plugin has
	// confirmed the volume capabilities, not that it hasn't
	// confirmed them, so if the field is nil we have to assume
	// the volume is ok.
	confirmedCaps := resp.GetConfirmed().GetVolumeCapabilities()

View on GitHub (pinned to 482b49bf1a)

Solutions

  1. Verify the volume still exists in the storage backend and correct the external volume ID in the volume spec / re-register the volume with nomad volume register using the correct ID.
  2. Check the CSI plugin's configuration (endpoint, secrets, region/cluster) to confirm it queries the storage account where the volume actually lives.
  3. If the volume was deleted intentionally, deregister the Nomad volume (nomad volume deregister) and clean up claims referencing it.
  4. Re-create the volume via ControllerCreateVolume if it should exist but was lost.

Example fix

// before
volume external_id = "vol-0abc123"  // deleted in cloud console
// nomad: volume could not be found

// after
$ nomad volume status vol-0abc123   # confirm failure
# re-register with the live provider volume ID
$ nomad volume register -volume-id=vol-0abc123 <updated-spec.hcl>
Defensive patterns

Strategy: try-catch

Validate before calling

ids, err := csi.ListVolumes(ctx)
if err != nil { return err }
valid := false
for _, v := range ids {
    if v.Id == volID { valid = true; break }
}
if !valid { return fmt.Errorf("volume %q not present on backend", volID) }

Try / catch

err := client.ControllerValidateCapabilities(ctx, req)
if err != nil {
    if strings.Contains(err.Error(), "could not be found") {
        // stop retrying; fix or re-register the volume ID
        return fmt.Errorf("permanent: %w", err)
    }
    return err
}

Prevention

When it happens

Trigger: Calling ControllerValidateCapabilities with an ExternalID that the storage provider does not recognize: the volume was deleted out-of-band, the ID was mistyped, the plugin is pointed at the wrong cluster/region, or the volume was never created/registered with that ID.

Common situations: A volume was deleted in the cloud console while Nomad still references its ID; volume registration points at a stale ID after switching storage backends; plugin misconfiguration (wrong secret/endpoint) makes it look up volumes in the wrong account; snapshot-restored or migrated environments where IDs changed.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c0b3c6a553f780db. Report an issue: GitHub.