hashicorp/nomad · error

error creating volume %q with plugin %q: %w: %s

Error message

error creating volume %q with plugin %q: %w: %s

What it means

Returned by HostVolumePluginExternal.Create when the plugin exits non-zero and DID return parseable JSON, allowing Nomad to append the plugin's own error message (pluginResp.Error) after the wrapped execution error. This is the richer variant of the create failure. Note the code warns that a plugin returning invalid JSON may leave a volume created without Nomad state.

Source

Thrown at client/hostvolumemanager/host_volume_plugin.go:363

		fmt.Sprintf("%s=%d", EnvCapacityMin, req.RequestedCapacityMinBytes),
		fmt.Sprintf("%s=%d", EnvCapacityMax, req.RequestedCapacityMaxBytes),
		fmt.Sprintf("%s=%s", EnvNodeID, req.NodeID),
		fmt.Sprintf("%s=%s", EnvParameters, params),
	}

	var pluginResp HostVolumePluginCreateResponse
	log := p.log.With("volume_name", req.Name, "volume_id", req.ID)
	stdout, _, err := p.runPlugin(ctx, log, "create", envVars)
	if err != nil {
		jsonErr := json.Unmarshal(stdout, &pluginResp)
		if jsonErr != nil {
			// if we got an error, we can't actually count on getting JSON, so
			// optimistically look for it and return the original error
			// otherwise
			return nil, fmt.Errorf(
				"error creating volume %q with plugin %q: %w", req.ID, p.ID, err)
		}
		return nil, fmt.Errorf("error creating volume %q with plugin %q: %w: %s",
			req.ID, p.ID, err, pluginResp.Error)
	}
	err = json.Unmarshal(stdout, &pluginResp)
	if err != nil {
		// note: if a plugin does not return valid json, a volume may be
		// created without any respective state in Nomad, since we return
		// an error here after the plugin has done who-knows-what.
		return nil, err
	}
	return &pluginResp, nil
}

// Delete calls the executable with the following parameters:
// arguments: $1=delete
// environment:
// - DHV_OPERATION=delete
// - DHV_CREATED_PATH={path that `create` returned}
// - DHV_VOLUMES_DIR={directory that volumes should be put in}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the trailing '%s' in the message — it contains the plugin's own error and act on it directly
2. Check permissions/existence of the plugin's VolumesDir on the client node
3. Verify volume ID uniqueness and parameter validity for this plugin
4. Run the plugin manually with NOMAD_OPERATION=create to reproduce
5. Reconcile Nomad state: a volume may have been created on disk without being registered

Example fix

// before
volume "web" { plugin = "mkdir" parameters {} }  # plugin error: mkdir: permission denied
// after
# on the client node
$ sudo mkdir -p /opt/nomad/volumes && sudo chown nomad:nomad /opt/nomad/volumes

if _, err := os.Stat(volumesDir); err != nil || !writable(volumesDir) {
    return fmt.Errorf("volumes dir %s not writable: %w", volumesDir, err)
}
if _, err := os.Stat(filepath.Join(volumesDir, volID)); err == nil {
    return fmt.Errorf("volume %s already exists", volID)
}

func pluginReportedError(msg string) bool {
    // message ends with ': <plugin error>' when JSON was parseable
    return strings.HasSuffix(msg, pluginErrMsg)
}

resp, err := plugin.Create(ctx, req)
if err != nil {
    parts := strings.Split(err.Error(), ": ")
    pluginMsg := parts[len(parts)-1]
    log.Error("volume create rejected by plugin", "plugin_error", pluginMsg)
    switch {
    case strings.Contains(pluginMsg, "permission denied"):
        // fix ownership and retry
    case strings.Contains(pluginMsg, "file exists"):
        // deregister/deduplicate ID first
    }
    return err
}

When it happens

Trigger: External create plugin exits non-zero but emits valid JSON containing an error field, e.g. it rejected the request: invalid volume ID, filesystem error creating the volume directory, permission denied in VolumesDir, or invalid parameters.

Common situations: Target volumes directory doesn't exist or lacks write permission; duplicate volume ID; plugin-side validation rejecting parameters; disk full; plugin compiled against a different response schema.

Related errors

• error creating volume %q with plugin %q: %w
• error reading plugin response when deleting volume %q with p
• error deleting volume %q with plugin %q: %w: %s
• error fingerprinting plugin %q: %w
• ErrPluginNotExists

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/fb7b92aba5957f58. Report an issue: GitHub.