hashicorp/nomad · error

failed executing plugin %q for secret %q: %w

Error message

failed executing plugin %q for secret %q: %w

What it means

This error wraps any failure from the external secret plugin's Fetch call inside Nomad's ExternalPluginProvider.Fetch. It means the plugin binary itself failed (crashed, non-zero exit, protocol error, timeout, or context cancellation) while retrieving the secret, and Nomad wraps the underlying cause with %w so the root error is preserved in the chain.

Source

Thrown at client/allocrunner/taskrunner/secrets/plugin_provider.go:55

	return &ExternalPluginProvider{
		plugin:     plugin,
		pluginName: pluginName,
		secretName: secretName,
		path:       path,
		env:        env,
	}
}

func (p *ExternalPluginProvider) InterpolateEnv(interpolate func(string) string) {
	for key, value := range p.env {
		p.env[key] = interpolate(value)
	}
}

func (p *ExternalPluginProvider) Fetch(ctx context.Context) (map[string]string, error) {
	resp, err := p.plugin.Fetch(ctx, p.path, p.env)
	if err != nil {
		return nil, fmt.Errorf("failed executing plugin %q for secret %q: %w", p.pluginName, p.secretName, err)
	}
	if resp.Error != nil {
		return nil, fmt.Errorf("provider %q for secret %q response contained error: %q", p.pluginName, p.secretName, *resp.Error)
	}

	formatted := make(map[string]string, len(resp.Result))
	for k, v := range resp.Result {
		formatted[fmt.Sprintf("secret.%s.%s", p.secretName, k)] = v
	}

	return formatted, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped root cause (%w) in the error chain — fix whatever the plugin itself reported (credentials, network, config).
2. Verify the plugin binary exists in the agent's plugin_dir, is executable, and matches the Nomad secrets-plugin protocol version.
3. Test the plugin standalone with its CLI/dev mode to reproduce and debug the failure.
4. Check agent logs for plugin lifecycle errors (launch, handshake, protocol mismatch).

Example fix

// before (plugin returns error due to missing CLOUD_TOKEN env)
resp, err := plugin.Fetch(ctx, path, env) // err: "missing CLOUD_TOKEN"
// after (ensure required env is passed to the plugin via the task env or agent config)
env["CLOUD_TOKEN"] = os.Getenv("CLOUD_TOKEN")
resp, err := plugin.Fetch(ctx, path, env)

// before scheduling the task, verify the plugin is deployable on the client
if _, err := os.Stat(filepath.Join(pluginDir, pluginName)); err != nil {
    return fmt.Errorf("secret plugin %s not present in plugin_dir: %w", pluginName, err)
}
if err := os.Chmod(filepath.Join(pluginDir, pluginName), 0o755); err != nil { /* not executable */ }

secrets, err := provider.Fetch(ctx)
if err != nil {
    var perr *PluginError
    if errors.As(err, &perr) {
        log.Error("secret plugin failed", "plugin", pluginName, "cause", errors.Unwrap(err))
        // retry with backoff or fail the task hook with the root cause
    }
    return err
}

When it happens

Trigger: Calling ExternalPluginProvider.Fetch (via the taskrunner secrets hook) when p.plugin.Fetch returns an error: plugin binary missing/not executable, plugin crashes or exits non-zero, plugin returns a malformed gRPC response, or ctx is cancelled/timed out during the fetch.

Common situations: Nomad client agents missing the vault-secrets or custom secret plugin binary in plugin_dir; plugin version incompatible with the Nomad API protocol; plugin lacking exec permissions after packaging; network/credential issues inside the plugin causing it to return an error; task shutdown cancelling the context mid-fetch.

Related errors

• provider %q for secret %q response contained error: %q
• %w: %q (wraps ErrPluginNotExists)
• %w: %q (wraps ErrPluginNotExecutable)
• namespace cannot contain template delimiters or parenthesis
• plugin not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/de145733d870062a. Report an issue: GitHub.