hashicorp/nomad · error

error querying volume %q: %v

Error message

error querying volume %q: %v

What it means

CreateSnapshot fails to look up the source volume in Nomad's state store via CSIVolumeByID and appends 'error querying volume %q: %v' to the multierror, continuing with the remaining snapshots. The inner error is a state-store query failure (RPC/connection error to the state backend or a store-level failure), not merely 'volume absent' — an absent volume takes the separate 'no such volume' path (vol == nil).

Source

Thrown at nomad/csi_endpoint.go:1619

		return structs.ErrPermissionDenied
	}

	state, err := v.srv.fsm.State().Snapshot()
	if err != nil {
		return err
	}

	method := "ClientCSI.ControllerCreateSnapshot"
	var mErr multierror.Error
	for _, snap := range args.Snapshots {
		if snap == nil {
			// we intentionally don't multierror here because we're in a weird state
			return fmt.Errorf("snapshot cannot be nil")
		}

		vol, err := state.CSIVolumeByID(nil, args.RequestNamespace(), snap.SourceVolumeID)
		if err != nil {
			multierror.Append(&mErr, fmt.Errorf("error querying volume %q: %v", snap.SourceVolumeID, err))
			continue
		}
		if vol == nil {
			multierror.Append(&mErr, fmt.Errorf("no such volume %q", snap.SourceVolumeID))
			continue
		}

		pluginID := snap.PluginID
		if pluginID == "" {
			pluginID = vol.PluginID
		}

		plugin, err := state.CSIPluginByID(nil, pluginID)
		if err != nil {
			multierror.Append(&mErr,
				fmt.Errorf("error querying plugin %q: %v", pluginID, err))
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v inner error in the multierror to identify the actual state-store failure and fix that root cause (server health, raft state, logs around the CSIVolumeByID call)
2. Retry CreateSnapshot once the server/state store is healthy — state query errors are frequently transient
3. Verify the request's RequestNamespace() is valid and reachable for the caller's ACL token; adjust the token's namespace rules
4. Collect and inspect all entries of the returned multierror.Error — some snapshots may have succeeded/failed independently

Example fix

// before
mErr := nomadCreateSnapshot(...) // opaque handling of multierror
// after
if err != nil { for _, e := range err.(*multierror.Error).Errors { log.Printf("snapshot error: %v", e) } } // then fix the underlying state-store cause per entry

vol, _, err := client.Volumes().Get(ctx, snap.SourceVolumeID, nil)
if err != nil { return fmt.Errorf("cannot snapshot: source volume %q not queryable: %w", snap.SourceVolumeID, err) }

err := createSnapshot(req)
if me, ok := err.(*multierror.Error); ok {
    for _, e := range me.Errors {
        if strings.Contains(e.Error(), "error querying volume") {
            // inspect wrapped cause; retry transient state-store failures with backoff
        }
    }
}

When it happens

Trigger: Sending CSISnapshotCreateRequest with Snapshots whose SourceVolumeID causes CSIVolumeByID(nil, ns, id) to return a non-nil error: state store backend errors, ACL-driven or namespace query failures surfaced as errors by the store, or a server-internal RPC error while reading volumes.

Common situations: Nomad raft/state store issues (server instability, corrupted state) during snapshot creation; querying with a namespace the store errors on rather than returning empty; transient server-side errors during multi-snapshot batch requests where later entries continue to be processed.

Related errors

• error parsing: root should be an object
• cannot specify Accessor ID
• volume snapshot ID cannot be updated
• failed to read dynamic plugin registry state: %v
• controller create snapshot: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/24ec3a3ba0aefc9c. Report an issue: GitHub.