{"record":{"id":"ead0876701815d9a","repo":"hashicorp/nomad","slug":"error-deleting-volume-q-with-plugin-q-w-s","errorCode":null,"errorMessage":"error deleting volume %q with plugin %q: %w: %s","messagePattern":"error deleting volume %q with plugin %q: %w: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/hostvolumemanager/host_volume_plugin.go","lineNumber":427,"sourceCode":"\t\tfmt.Sprintf(\"%s=%s\", EnvNamespace, req.Namespace),\n\t\tfmt.Sprintf(\"%s=%s\", EnvVolumeName, req.Name),\n\t\tfmt.Sprintf(\"%s=%s\", EnvVolumeID, req.ID),\n\t\tfmt.Sprintf(\"%s=%s\", EnvNodeID, req.NodeID),\n\t\tfmt.Sprintf(\"%s=%s\", EnvParameters, params),\n\t}\n\n\tlog := p.log.With(\"volume_name\", req.Name, \"volume_id\", req.ID)\n\tstdout, _, err := p.runPlugin(ctx, log, \"delete\", envVars)\n\tif err != nil {\n\t\tvar pluginResp HostVolumePluginDeleteResponse\n\t\tjsonErr := json.Unmarshal(stdout, &pluginResp)\n\t\tif jsonErr != nil {\n\t\t\t// if we got an error, we can't actually count on getting JSON, so\n\t\t\t// optimistically look for it and return the original error\n\t\t\t// otherwise\n\t\t\treturn fmt.Errorf(\"error reading plugin response when deleting volume %q with plugin %q: original error: %w\", req.ID, p.ID, err)\n\t\t}\n\t\treturn fmt.Errorf(\"error deleting volume %q with plugin %q: %w: %s\",\n\t\t\treq.ID, p.ID, err, pluginResp.Error)\n\t}\n\n\treturn nil\n}\n\n// runPlugin executes the... executable\nfunc (p *HostVolumePluginExternal) runPlugin(ctx context.Context, log hclog.Logger,\n\top string, env []string) (stdout, stderr []byte, err error) {\n\n\tlog = log.With(\"operation\", op)\n\tlog.Debug(\"running plugin\")\n\n\t// set up plugin execution\n\tcmd := exec.CommandContext(ctx, p.Executable, op)\n\tcmd.Env = env\n\n\tstdout, stderr, err = runCommand(cmd)","sourceCodeStart":409,"sourceCodeEnd":445,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/hostvolumemanager/host_volume_plugin.go#L409-L445","documentation":"Returned by HostVolumePluginExternal.Delete when the plugin exits non-zero and returned valid JSON, so Nomad can append the plugin's own error message after the wrapped execution error. It is the informative variant of a delete failure and names both the volume ID and plugin ID.","triggerScenarios":"External delete plugin exits non-zero with parseable JSON output, e.g. volume directory not found, permission denied removing the directory, or plugin-side validation failure.","commonSituations":"Volume already deleted out-of-band (double delete); VolumesDir permissions changed; disk/filesystem errors; plugin rejects unknown volume ID; plugin schema mismatch causing unexpected Error field values.","solutions":["Read the plugin's error text at the end of the message and fix that condition","If the volume is already gone on disk, treat the delete as complete and deregister in Nomad","Fix ownership/permissions on the volume directory under VolumesDir","Run the plugin manually with NOMAD_OPERATION=delete to reproduce","Ensure only Nomad manages volumes in that directory to avoid out-of-band deletions"],"exampleFix":"// before\n$ rm -rf /opt/nomad/volumes/web  # out-of-band delete\n$ nomad volume deregister web    # plugin error: no such volume\n// after\n$ nomad volume deregister web    # let Nomad invoke the plugin delete itself","handlingStrategy":"try-catch","validationCode":"if _, err := os.Stat(filepath.Join(volumesDir, volID)); os.IsNotExist(err) {\n    return nil // already deleted; skip plugin call\n}","typeGuard":"func isPluginDeleteError(err error) bool {\n    return strings.Contains(err.Error(), \"error deleting volume\")\n}","tryCatchPattern":"err := plugin.Delete(ctx, req)\nif err != nil {\n    if isPluginDeleteError(err) {\n        if strings.Contains(err.Error(), \"no such file or directory\") {\n            return nil // treat as idempotent success\n        }\n        return fmt.Errorf(\"plugin rejected delete: %w\", err)\n    }\n    return err\n}","preventionTips":["Treat delete as idempotent — tolerate 'not found' plugin errors","Do not remove volume directories out-of-band","Keep VolumesDir ownership stable for the nomad user","Read the appended plugin error text to pick the right remediation"],"tags":["host-volumes","external-plugin","process-execution"],"backgroundTag":"plugin-execution-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}