{"record":{"id":"9c49a3d7456613f4","repo":"hashicorp/nomad","slug":"could-not-create-snapshot-v","errorCode":null,"errorMessage":"could not create snapshot: %v","messagePattern":"could not create snapshot: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/csi_endpoint.go","lineNumber":1664,"sourceCode":"\t\t}\n\n\t\tsecrets := vol.Secrets\n\t\t// merge request secrets onto volume secrets\n\t\tmaps.Copy(secrets, snap.Secrets)\n\n\t\tcReq := &cstructs.ClientCSIControllerCreateSnapshotRequest{\n\t\t\tExternalSourceVolumeID: vol.ExternalID,\n\t\t\tName:                   snap.Name,\n\t\t\tSecrets:                secrets,\n\t\t\tParameters:             snap.Parameters,\n\t\t}\n\t\tcReq.PluginID = pluginID\n\t\tcResp := &cstructs.ClientCSIControllerCreateSnapshotResponse{}\n\t\terr = v.serializedControllerRPC(pluginID, func() error {\n\t\t\treturn v.srv.RPC(method, cReq, cResp)\n\t\t})\n\t\tif err != nil {\n\t\t\tmultierror.Append(&mErr, fmt.Errorf(\"could not create snapshot: %v\", err))\n\t\t\tcontinue\n\t\t}\n\t\treply.Snapshots = append(reply.Snapshots, &structs.CSISnapshot{\n\t\t\tID:                     cResp.ID,\n\t\t\tExternalSourceVolumeID: cResp.ExternalSourceVolumeID,\n\t\t\tSizeBytes:              cResp.SizeBytes,\n\t\t\tCreateTime:             cResp.CreateTime,\n\t\t\tIsReady:                cResp.IsReady,\n\t\t})\n\t}\n\n\treturn mErr.ErrorOrNil()\n}\n\nfunc (v *CSIVolume) DeleteSnapshot(args *structs.CSISnapshotDeleteRequest, reply *structs.CSISnapshotDeleteResponse) error {\n\n\tauthErr := v.srv.Authenticate(v.ctx, args)\n\tif done, err := v.srv.forward(\"CSIVolume.DeleteSnapshot\", args, args, reply); done {","sourceCodeStart":1646,"sourceCodeEnd":1682,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L1646-L1682","documentation":"After validation, CreateSnapshot forwards the request to the CSI controller plugin on a client node via serializedControllerRPC and the server RPC method. Any error returned by the plugin/controller (unreachable node, plugin RPC failure, storage backend error) is wrapped as 'could not create snapshot: %v' and appended to the multierror.","triggerScenarios":"The CSI controller RPC fails: client node running the controller is down/drained, plugin task crashed, serializedControllerRPC cannot find a running controller, or the storage backend rejects the CreateSnapshot call (quota, missing source volume, auth/secret mismatch).","commonSituations":"Node drain or maintenance during snapshot; wrong volume secrets merged into the request; storage backend quota exceeded; controller plugin job failed; network partition between server and client.","solutions":["Read the wrapped %v detail for the root cause and check the client node logs for the CSI plugin task","Confirm the controller plugin allocation is running ('nomad job status <csi-plugin-job>') and the node is eligible; reschedule if drained","Verify volume secrets are correct — request secrets are merged over volume secrets and bad credentials fail at the storage backend","Retry the snapshot once the controller is reachable"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// confirm controller allocation is running before snapshotting\nallocs, _ := client.Jobs().Allocations(csiPluginJob, false)\nrunning := false\nfor _, a := range allocs { if a.ClientStatus == \"running\" { running = true } }\nif !running { return errors.New(\"CSI controller not running\") }","typeGuard":null,"tryCatchPattern":"err := client.Volumes().CreateSnapshot(req)\nif err != nil && strings.Contains(err.Error(), \"could not create snapshot\") {\n    // log wrapped cause; retry with backoff if transient (node unreachable)\n}","preventionTips":["Monitor CSI plugin job health and node eligibility","Ensure volume secrets are current before snapshotting","Avoid snapshots during node drains/maintenance windows","Read the wrapped %v detail to distinguish transient vs permanent backend failures"],"tags":["nomad","csi","rpc","snapshot","storage-backend"],"backgroundTag":"csi-controller-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}