hashicorp/nomad · error

error reading plugin response when deleting volume %q with p

Error message

error reading plugin response when deleting volume %q with plugin %q: original error: %w

What it means

Returned by HostVolumePluginExternal.Delete when the external plugin exits non-zero AND its stdout cannot be parsed as JSON, so the plugin's structured error is unavailable and only the original execution error is wrapped ('original error: %w'). It tells you the delete failed but the plugin gave no machine-readable reason.

Source

Thrown at client/hostvolumemanager/host_volume_plugin.go:425

		fmt.Sprintf("%s=%s", EnvCreatedPath, req.HostPath),
		// values from volume spec
		fmt.Sprintf("%s=%s", EnvNamespace, req.Namespace),
		fmt.Sprintf("%s=%s", EnvVolumeName, req.Name),
		fmt.Sprintf("%s=%s", EnvVolumeID, req.ID),
		fmt.Sprintf("%s=%s", EnvNodeID, req.NodeID),
		fmt.Sprintf("%s=%s", EnvParameters, params),
	}

	log := p.log.With("volume_name", req.Name, "volume_id", req.ID)
	stdout, _, err := p.runPlugin(ctx, log, "delete", envVars)
	if err != nil {
		var pluginResp HostVolumePluginDeleteResponse
		jsonErr := json.Unmarshal(stdout, &pluginResp)
		if jsonErr != nil {
			// if we got an error, we can't actually count on getting JSON, so
			// optimistically look for it and return the original error
			// otherwise
			return fmt.Errorf("error reading plugin response when deleting volume %q with plugin %q: original error: %w", req.ID, p.ID, err)
		}
		return fmt.Errorf("error deleting volume %q with plugin %q: %w: %s",
			req.ID, p.ID, err, pluginResp.Error)
	}

	return nil
}

// runPlugin executes the... executable
func (p *HostVolumePluginExternal) runPlugin(ctx context.Context, log hclog.Logger,
	op string, env []string) (stdout, stderr []byte, err error) {

	log = log.With("operation", op)
	log.Debug("running plugin")

	// set up plugin execution
	cmd := exec.CommandContext(ctx, p.Executable, op)
	cmd.Env = env

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Execute the plugin manually with NOMAD_OPERATION=delete and the same env to see the raw failure
2. Inspect Nomad client logs for the plugin's stderr
3. Verify the plugin binary exists, is executable, and is the intended one at PluginDir
4. Make the plugin emit JSON ({"error": ...}) on failure paths
5. Check whether the volume directory still exists / permission to delete it

Example fix

// before
$ /opt/nomad-plugins/delete.sh  # segfault, no stdout
// after
$ file /opt/nomad-plugins/delete.sh && chmod +x /opt/nomad-plugins/delete.sh
$ # rebuild plugin to print {"error": "volume not found"} instead of crashing

fi, err := os.Stat(pluginPath)
if err != nil || fi.Mode()&0o111 == 0 {
    return fmt.Errorf("delete plugin %s missing or not executable", pluginPath)
}

func isDeleteNoJsonErr(err error) bool {
    return strings.Contains(err.Error(), "error reading plugin response when deleting volume")
}

err := plugin.Delete(ctx, req)
if err != nil {
    if isDeleteNoJsonErr(err) {
        // plugin failed without JSON; check client logs for stderr,
        // then verify whether the volume dir still exists before retrying
        return fmt.Errorf("delete plugin produced no JSON: %w", err)
    }
    return err
}

When it happens

Trigger: Delete invocation of the plugin binary exits non-zero without valid JSON on stdout — plugin crash, missing binary/interpreter, or failure before it writes a response.

Common situations: Plugin script not executable or bad shebang; plugin crashing on missing volume; errors only on stderr; wrong PluginDir configured; plugin version mismatch with Nomad's expected response schema.

Related errors

• error creating volume %q with plugin %q: %w
• error creating volume %q with plugin %q: %w: %s
• error deleting volume %q with plugin %q: %w: %s
• error fingerprinting plugin %q: %w
• ErrPluginNotExists

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/777298f5bba94c7d. Report an issue: GitHub.