hashicorp/nomad · error

invalid starting token %q: %v

Error message

invalid starting token %q: %v

What it means

ControllerListVolumes got gRPC code Aborted from the CSI plugin, which per the CSI spec means the pagination StartingToken is invalid — expired, malformed, or belonging to a different enumeration session. Nomad wraps it as "invalid starting token" using req.StartingToken.

Source

Thrown at plugins/csi/client.go:482

	return NewCreateVolumeResponse(resp), nil
}

func (c *client) ControllerListVolumes(ctx context.Context, req *ControllerListVolumesRequest, opts ...grpc.CallOption) (*ControllerListVolumesResponse, error) {
	if err := c.ensureConnected(ctx); err != nil {
		return nil, err
	}

	err := req.Validate()
	if err != nil {
		return nil, err
	}
	creq := req.ToCSIRepresentation()
	resp, err := c.controllerClient.ListVolumes(ctx, creq, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.Aborted:
			return nil, fmt.Errorf(
				"invalid starting token %q: %v", req.StartingToken, err)
		case codes.Internal:
			return nil, fmt.Errorf(
				"controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return nil, err
	}
	return NewListVolumesResponse(resp), nil
}

func (c *client) ControllerDeleteVolume(ctx context.Context, req *ControllerDeleteVolumeRequest, opts ...grpc.CallOption) error {
	if err := c.ensureConnected(ctx); err != nil {
		return err
	}

	err := req.Validate()
	if err != nil {
		return err

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the list without a starting token to enumerate volumes from the beginning.
2. Re-run the pagination loop in one pass without long delays between pages.
3. If pagination repeatedly fails, list all pages and paginate client-side instead of persisting tokens.
4. Check whether the plugin restarted between calls; restart your pagination with a fresh token.

Example fix

// before (reusing stale token)
resp, err := client.ListVolumes(ctx, &csi.ListVolumesRequest{StartingToken: oldToken})
// after (fresh listing)
resp, err := client.ListVolumes(ctx, &csi.ListVolumesRequest{})

// Don't persist tokens across plugin restarts or long periods
if tokenAge(startingToken) > maxTokenTTL {
    startingToken = "" // restart listing from the first page
}

resp, err := client.ControllerListVolumes(ctx, req)
if err != nil && strings.Contains(err.Error(), "invalid starting token") {
    req.StartingToken = ""
    resp, err = client.ControllerListVolumes(ctx, req) // restart from page 1
}

When it happens

Trigger: Calling ControllerListVolumes with a request whose StartingToken is stale or corrupted — e.g. resuming a listing after the plugin restarted, using a token from a different plugin, or a client that cached the token too long.

Common situations: Long-running external tool paginating volumes with delays between pages; plugin restart or failover invalidated in-memory pagination state; token copy/paste error; mixing plugins (token from vendor A used against vendor B).

Related errors

• CSI.ControllerListVolumes: %v
• CSI.ControllerListSnapshots: %v
• nil response from plugin.NodeExpandVolume
• MaxEntries cannot be negative
• CSI plugin failed to register: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f2f228444585d08a. Report an issue: GitHub.