hashicorp/nomad · error

error creating volume %q with plugin %q: %w

Error message

error creating volume %q with plugin %q: %w

What it means

Returned by HostVolumePluginExternal.Create when the external plugin executable exits non-zero AND its stdout could not be parsed as JSON, so Nomad wraps the original execution error with the volume ID and plugin ID. Because no structured response is available, only the original error is propagated (no pluginResp.Error detail).

Source

Thrown at client/hostvolumemanager/host_volume_plugin.go:360

		fmt.Sprintf("%s=%s", EnvNamespace, req.Namespace),
		fmt.Sprintf("%s=%s", EnvVolumeName, req.Name),
		fmt.Sprintf("%s=%s", EnvVolumeID, req.ID),
		fmt.Sprintf("%s=%d", EnvCapacityMin, req.RequestedCapacityMinBytes),
		fmt.Sprintf("%s=%d", EnvCapacityMax, req.RequestedCapacityMaxBytes),
		fmt.Sprintf("%s=%s", EnvNodeID, req.NodeID),
		fmt.Sprintf("%s=%s", EnvParameters, params),
	}

	var pluginResp HostVolumePluginCreateResponse
	log := p.log.With("volume_name", req.Name, "volume_id", req.ID)
	stdout, _, err := p.runPlugin(ctx, log, "create", envVars)
	if err != nil {
		jsonErr := json.Unmarshal(stdout, &pluginResp)
		if jsonErr != nil {
			// if we got an error, we can't actually count on getting JSON, so
			// optimistically look for it and return the original error
			// otherwise
			return nil, fmt.Errorf(
				"error creating volume %q with plugin %q: %w", req.ID, p.ID, err)
		}
		return nil, fmt.Errorf("error creating volume %q with plugin %q: %w: %s",
			req.ID, p.ID, err, pluginResp.Error)
	}
	err = json.Unmarshal(stdout, &pluginResp)
	if err != nil {
		// note: if a plugin does not return valid json, a volume may be
		// created without any respective state in Nomad, since we return
		// an error here after the plugin has done who-knows-what.
		return nil, err
	}
	return &pluginResp, nil
}

// Delete calls the executable with the following parameters:
// arguments: $1=delete
// environment:

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Run the plugin manually with the same env vars (NOMAD_OPERATION=create etc.) to reproduce the failure
2. Check the plugin's stderr output in the Nomad client logs for the root cause
3. Verify p.ID and p.PluginDir point at the intended executable and it is executable
4. Fix or reinstall the plugin script so it emits JSON even on error
5. Validate req.ID / parameters against the plugin's expected schema

Example fix

// before
$ /opt/nomad-plugins/create.sh  # crashes, no JSON on stdout
// after
$ chmod +x /opt/nomad-plugins/create.sh && head -1 create.sh  # ensure '#!/bin/sh'
$ # fix script to output {"error": "..."} on failure

fi, err := os.Stat(pluginPath)
if err != nil || fi.Mode()&0o111 == 0 {
    return fmt.Errorf("plugin %s missing or not executable", pluginPath)
}

func isPluginExecErr(err error) bool {
    return strings.Contains(err.Error(), "with plugin") && !strings.Contains(err.Error(), ": ") // no pluginResp suffix
}

resp, err := plugin.Create(ctx, req)
if err != nil {
    var perr *SomeRetryable = nil
    if errors.As(err, &perr) { /* retry */ }
    log.Error("plugin create failed without JSON response; check plugin stderr in client logs", "err", err)
    return err
}

When it happens

Trigger: The plugin binary invoked by Create fails (non-zero exit) and prints no valid JSON on stdout — e.g. crash, missing interpreter, hard failure before emitting a response.

Common situations: Plugin executable missing/shebang broken; plugin crashing on bad parameters; plugin writing errors only to stderr; wrong plugin directory configured so an incompatible script runs; script has a bug that skips JSON output on failure.

Related errors

• error creating volume %q with plugin %q: %w: %s
• error reading plugin response when deleting volume %q with p
• error deleting volume %q with plugin %q: %w: %s
• error fingerprinting plugin %q: %w
• ErrPluginNotExists

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/792589d439d7ff56. Report an issue: GitHub.