hashicorp/nomad · error

error deleting volume %q with plugin %q: %w: %s

Error message

error deleting volume %q with plugin %q: %w: %s

What it means

Returned by HostVolumePluginExternal.Delete when the plugin exits non-zero and returned valid JSON, so Nomad can append the plugin's own error message after the wrapped execution error. It is the informative variant of a delete failure and names both the volume ID and plugin ID.

Source

Thrown at client/hostvolumemanager/host_volume_plugin.go:427

		fmt.Sprintf("%s=%s", EnvNamespace, req.Namespace),
		fmt.Sprintf("%s=%s", EnvVolumeName, req.Name),
		fmt.Sprintf("%s=%s", EnvVolumeID, req.ID),
		fmt.Sprintf("%s=%s", EnvNodeID, req.NodeID),
		fmt.Sprintf("%s=%s", EnvParameters, params),
	}

	log := p.log.With("volume_name", req.Name, "volume_id", req.ID)
	stdout, _, err := p.runPlugin(ctx, log, "delete", envVars)
	if err != nil {
		var pluginResp HostVolumePluginDeleteResponse
		jsonErr := json.Unmarshal(stdout, &pluginResp)
		if jsonErr != nil {
			// if we got an error, we can't actually count on getting JSON, so
			// optimistically look for it and return the original error
			// otherwise
			return fmt.Errorf("error reading plugin response when deleting volume %q with plugin %q: original error: %w", req.ID, p.ID, err)
		}
		return fmt.Errorf("error deleting volume %q with plugin %q: %w: %s",
			req.ID, p.ID, err, pluginResp.Error)
	}

	return nil
}

// runPlugin executes the... executable
func (p *HostVolumePluginExternal) runPlugin(ctx context.Context, log hclog.Logger,
	op string, env []string) (stdout, stderr []byte, err error) {

	log = log.With("operation", op)
	log.Debug("running plugin")

	// set up plugin execution
	cmd := exec.CommandContext(ctx, p.Executable, op)
	cmd.Env = env

	stdout, stderr, err = runCommand(cmd)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the plugin's error text at the end of the message and fix that condition
2. If the volume is already gone on disk, treat the delete as complete and deregister in Nomad
3. Fix ownership/permissions on the volume directory under VolumesDir
4. Run the plugin manually with NOMAD_OPERATION=delete to reproduce
5. Ensure only Nomad manages volumes in that directory to avoid out-of-band deletions

Example fix

// before
$ rm -rf /opt/nomad/volumes/web  # out-of-band delete
$ nomad volume deregister web    # plugin error: no such volume
// after
$ nomad volume deregister web    # let Nomad invoke the plugin delete itself

if _, err := os.Stat(filepath.Join(volumesDir, volID)); os.IsNotExist(err) {
    return nil // already deleted; skip plugin call
}

func isPluginDeleteError(err error) bool {
    return strings.Contains(err.Error(), "error deleting volume")
}

err := plugin.Delete(ctx, req)
if err != nil {
    if isPluginDeleteError(err) {
        if strings.Contains(err.Error(), "no such file or directory") {
            return nil // treat as idempotent success
        }
        return fmt.Errorf("plugin rejected delete: %w", err)
    }
    return err
}

When it happens

Trigger: External delete plugin exits non-zero with parseable JSON output, e.g. volume directory not found, permission denied removing the directory, or plugin-side validation failure.

Common situations: Volume already deleted out-of-band (double delete); VolumesDir permissions changed; disk/filesystem errors; plugin rejects unknown volume ID; plugin schema mismatch causing unexpected Error field values.

Related errors

• error creating volume %q with plugin %q: %w
• error creating volume %q with plugin %q: %w: %s
• error reading plugin response when deleting volume %q with p
• error fingerprinting plugin %q: %w
• ErrPluginNotExists

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ead0876701815d9a. Report an issue: GitHub.