hashicorp/nomad · error

could not create snapshot: %v

Error message

could not create snapshot: %v

What it means

After validation, CreateSnapshot forwards the request to the CSI controller plugin on a client node via serializedControllerRPC and the server RPC method. Any error returned by the plugin/controller (unreachable node, plugin RPC failure, storage backend error) is wrapped as 'could not create snapshot: %v' and appended to the multierror.

Source

Thrown at nomad/csi_endpoint.go:1664

		}

		secrets := vol.Secrets
		// merge request secrets onto volume secrets
		maps.Copy(secrets, snap.Secrets)

		cReq := &cstructs.ClientCSIControllerCreateSnapshotRequest{
			ExternalSourceVolumeID: vol.ExternalID,
			Name:                   snap.Name,
			Secrets:                secrets,
			Parameters:             snap.Parameters,
		}
		cReq.PluginID = pluginID
		cResp := &cstructs.ClientCSIControllerCreateSnapshotResponse{}
		err = v.serializedControllerRPC(pluginID, func() error {
			return v.srv.RPC(method, cReq, cResp)
		})
		if err != nil {
			multierror.Append(&mErr, fmt.Errorf("could not create snapshot: %v", err))
			continue
		}
		reply.Snapshots = append(reply.Snapshots, &structs.CSISnapshot{
			ID:                     cResp.ID,
			ExternalSourceVolumeID: cResp.ExternalSourceVolumeID,
			SizeBytes:              cResp.SizeBytes,
			CreateTime:             cResp.CreateTime,
			IsReady:                cResp.IsReady,
		})
	}

	return mErr.ErrorOrNil()
}

func (v *CSIVolume) DeleteSnapshot(args *structs.CSISnapshotDeleteRequest, reply *structs.CSISnapshotDeleteResponse) error {

	authErr := v.srv.Authenticate(v.ctx, args)
	if done, err := v.srv.forward("CSIVolume.DeleteSnapshot", args, args, reply); done {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v detail for the root cause and check the client node logs for the CSI plugin task
2. Confirm the controller plugin allocation is running ('nomad job status <csi-plugin-job>') and the node is eligible; reschedule if drained
3. Verify volume secrets are correct — request secrets are merged over volume secrets and bad credentials fail at the storage backend
4. Retry the snapshot once the controller is reachable

// confirm controller allocation is running before snapshotting
allocs, _ := client.Jobs().Allocations(csiPluginJob, false)
running := false
for _, a := range allocs { if a.ClientStatus == "running" { running = true } }
if !running { return errors.New("CSI controller not running") }

err := client.Volumes().CreateSnapshot(req)
if err != nil && strings.Contains(err.Error(), "could not create snapshot") {
    // log wrapped cause; retry with backoff if transient (node unreachable)
}

When it happens

Trigger: The CSI controller RPC fails: client node running the controller is down/drained, plugin task crashed, serializedControllerRPC cannot find a running controller, or the storage backend rejects the CreateSnapshot call (quota, missing source volume, auth/secret mismatch).

Common situations: Node drain or maintenance during snapshot; wrong volume secrets merged into the request; storage backend quota exceeded; controller plugin job failed; network partition between server and client.

Related errors

• volume snapshot ID cannot be updated
• could not claim volume %s: %w
• Unexpected nil volume returned for ID: %v
• controller create snapshot: %v
• controller delete snapshot: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9c49a3d7456613f4. Report an issue: GitHub.