{"record":{"id":"777298f5bba94c7d","repo":"hashicorp/nomad","slug":"error-reading-plugin-response-when-deleting-volume","errorCode":null,"errorMessage":"error reading plugin response when deleting volume %q with plugin %q: original error: %w","messagePattern":"error reading plugin response when deleting volume %q with plugin %q: original error: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/hostvolumemanager/host_volume_plugin.go","lineNumber":425,"sourceCode":"\t\tfmt.Sprintf(\"%s=%s\", EnvCreatedPath, req.HostPath),\n\t\t// values from volume spec\n\t\tfmt.Sprintf(\"%s=%s\", EnvNamespace, req.Namespace),\n\t\tfmt.Sprintf(\"%s=%s\", EnvVolumeName, req.Name),\n\t\tfmt.Sprintf(\"%s=%s\", EnvVolumeID, req.ID),\n\t\tfmt.Sprintf(\"%s=%s\", EnvNodeID, req.NodeID),\n\t\tfmt.Sprintf(\"%s=%s\", EnvParameters, params),\n\t}\n\n\tlog := p.log.With(\"volume_name\", req.Name, \"volume_id\", req.ID)\n\tstdout, _, err := p.runPlugin(ctx, log, \"delete\", envVars)\n\tif err != nil {\n\t\tvar pluginResp HostVolumePluginDeleteResponse\n\t\tjsonErr := json.Unmarshal(stdout, &pluginResp)\n\t\tif jsonErr != nil {\n\t\t\t// if we got an error, we can't actually count on getting JSON, so\n\t\t\t// optimistically look for it and return the original error\n\t\t\t// otherwise\n\t\t\treturn fmt.Errorf(\"error reading plugin response when deleting volume %q with plugin %q: original error: %w\", req.ID, p.ID, err)\n\t\t}\n\t\treturn fmt.Errorf(\"error deleting volume %q with plugin %q: %w: %s\",\n\t\t\treq.ID, p.ID, err, pluginResp.Error)\n\t}\n\n\treturn nil\n}\n\n// runPlugin executes the... executable\nfunc (p *HostVolumePluginExternal) runPlugin(ctx context.Context, log hclog.Logger,\n\top string, env []string) (stdout, stderr []byte, err error) {\n\n\tlog = log.With(\"operation\", op)\n\tlog.Debug(\"running plugin\")\n\n\t// set up plugin execution\n\tcmd := exec.CommandContext(ctx, p.Executable, op)\n\tcmd.Env = env","sourceCodeStart":407,"sourceCodeEnd":443,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/hostvolumemanager/host_volume_plugin.go#L407-L443","documentation":"Returned by HostVolumePluginExternal.Delete when the external plugin exits non-zero AND its stdout cannot be parsed as JSON, so the plugin's structured error is unavailable and only the original execution error is wrapped ('original error: %w'). It tells you the delete failed but the plugin gave no machine-readable reason.","triggerScenarios":"Delete invocation of the plugin binary exits non-zero without valid JSON on stdout — plugin crash, missing binary/interpreter, or failure before it writes a response.","commonSituations":"Plugin script not executable or bad shebang; plugin crashing on missing volume; errors only on stderr; wrong PluginDir configured; plugin version mismatch with Nomad's expected response schema.","solutions":["Execute the plugin manually with NOMAD_OPERATION=delete and the same env to see the raw failure","Inspect Nomad client logs for the plugin's stderr","Verify the plugin binary exists, is executable, and is the intended one at PluginDir","Make the plugin emit JSON ({\"error\": ...}) on failure paths","Check whether the volume directory still exists / permission to delete it"],"exampleFix":"// before\n$ /opt/nomad-plugins/delete.sh  # segfault, no stdout\n// after\n$ file /opt/nomad-plugins/delete.sh && chmod +x /opt/nomad-plugins/delete.sh\n$ # rebuild plugin to print {\"error\": \"volume not found\"} instead of crashing","handlingStrategy":"try-catch","validationCode":"fi, err := os.Stat(pluginPath)\nif err != nil || fi.Mode()&0o111 == 0 {\n    return fmt.Errorf(\"delete plugin %s missing or not executable\", pluginPath)\n}","typeGuard":"func isDeleteNoJsonErr(err error) bool {\n    return strings.Contains(err.Error(), \"error reading plugin response when deleting volume\")\n}","tryCatchPattern":"err := plugin.Delete(ctx, req)\nif err != nil {\n    if isDeleteNoJsonErr(err) {\n        // plugin failed without JSON; check client logs for stderr,\n        // then verify whether the volume dir still exists before retrying\n        return fmt.Errorf(\"delete plugin produced no JSON: %w\", err)\n    }\n    return err\n}","preventionTips":["Test plugin delete path manually with NOMAD_OPERATION=delete","Ensure plugin emits JSON on all failure paths","Verify plugin executability and shebang before deployment","Check that the volume directory exists and is removable before calling Delete"],"tags":["host-volumes","external-plugin","process-execution"],"backgroundTag":"plugin-execution-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}