{"record":{"id":"6e3a988378b5f5f2","repo":"hashicorp/nomad","slug":"v-v","errorCode":null,"errorMessage":"%v: %v","messagePattern":"%v: %v","errorType":"exception","errorClass":"ErrCSIClientRPCRetryable","httpStatus":null,"severity":"warning","filePath":"nomad/csi_endpoint.go","lineNumber":593,"sourceCode":"\tcReq := &cstructs.ClientCSIControllerAttachVolumeRequest{\n\t\tVolumeID:        vol.RemoteID(),\n\t\tClientCSINodeID: req.ExternalNodeID,\n\t\tAttachmentMode:  req.AttachmentMode,\n\t\tAccessMode:      req.AccessMode,\n\t\tMountOptions:    csiVolumeMountOptions(vol.MountOptions),\n\t\tReadOnly:        req.Claim == structs.CSIVolumeClaimRead,\n\t\tSecrets:         vol.Secrets,\n\t\tVolumeContext:   vol.Context,\n\t}\n\tcReq.PluginID = plug.ID\n\tcResp := &cstructs.ClientCSIControllerAttachVolumeResponse{}\n\n\terr = v.serializedControllerRPC(plug.ID, func() error {\n\t\treturn v.srv.RPC(method, cReq, cResp)\n\t})\n\tif err != nil {\n\t\tif strings.Contains(err.Error(), \"FailedPrecondition\") {\n\t\t\treturn fmt.Errorf(\"%v: %v\", structs.ErrCSIClientRPCRetryable, err)\n\t\t}\n\t\treturn err\n\t}\n\tresp.PublishContext = cResp.PublishContext\n\treturn nil\n}\n\nfunc (v *CSIVolume) volAndPluginLookup(namespace, volID string) (*structs.CSIPlugin, *structs.CSIVolume, error) {\n\tstate := v.srv.fsm.State()\n\tvol, err := state.CSIVolumeByID(nil, namespace, volID)\n\tif err != nil {\n\t\treturn nil, nil, err\n\t}\n\tif vol == nil {\n\t\treturn nil, nil, fmt.Errorf(\"volume not found: %s\", volID)\n\t}\n\tif !vol.ControllerRequired {\n\t\treturn nil, vol, nil","sourceCodeStart":575,"sourceCodeEnd":611,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L575-L611","documentation":"The CSI controller attach RPC (ClientCSI.ControllerAttachVolume) sent to the plugin returned an error containing \"FailedPrecondition\", so controllerPublishVolume wraps it with structs.ErrCSIClientRPCRetryable to mark it safe to retry. The plugin is telling Nomad the volume is in a state that doesn't allow attach yet (e.g. being detached, published elsewhere, or still creating).","triggerScenarios":"Raised when v.srv.RPC(\"ClientCSI.ControllerAttachVolume\", ...) returns an error whose message contains \"FailedPrecondition\" (a gRPC status string surfaced by the plugin). Common with plugins that serialize attach/detach per volume: the volume is mid-detach from a previous node, or a conflicting attach is in flight.","commonSituations":"Rapid rescheduling of a job across nodes while the old detach hasn't completed; EBS/EFS/Azure-disk plugins rejecting attach while the volume is still detaching from another instance; leadership transitions leaving the serialized controller RPC queue in a transient state.","solutions":["Retry the claim/operation after a short backoff — ErrCSIClientRPCRetryable signals the RPC is retryable and Nomad's internal retry loops will re-run it.","Check `nomad volume status <vol-id>` for conflicting claims; release stale claims from the old node if a detach is stuck.","Verify the plugin on the client is healthy and up to date; some plugins have known races in attach/detach serialization.","If it persists, restart the CSI plugin task on the client node to clear stuck controller state."],"exampleFix":"// caller-side: treat as retryable\nerr := client.CSIVolumes().Claim(volID, ns, opts, req)\nif err != nil && strings.Contains(err.Error(), \"FailedPrecondition\") {\n    time.Sleep(5 * time.Second) // retry with backoff\n    return retry()\n}","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"FailedPrecondition\") {\n    time.Sleep(backoff) // ErrCSIClientRPCRetryable: safe to retry\n    return retryClaim()\n}","preventionTips":["Avoid rapid reschedules across nodes while detaches are in flight.","Verify volume claims are released before rescheduling the job.","Keep CSI plugin versions current — many attach races are fixed upstream."],"tags":["nomad","csi","grpc","retryable","attach"],"backgroundTag":"grpc-failed-precondition","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}